Sep 16 13:41:02.413133 ip-10-0-143-222 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 13:41:02.413142 ip-10-0-143-222 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 13:41:02.413149 ip-10-0-143-222 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 13:41:02.413294 ip-10-0-143-222 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 13:41:12.633348 ip-10-0-143-222 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 13:41:12.633367 ip-10-0-143-222 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bfdbfdd8a2a24588bf4a35450c32a3de -- Sep 16 13:42:45.910460 ip-10-0-143-222 systemd[1]: Starting Kubernetes Kubelet... Sep 16 13:42:46.353334 ip-10-0-143-222 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:42:46.353334 ip-10-0-143-222 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 13:42:46.353334 ip-10-0-143-222 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:42:46.353334 ip-10-0-143-222 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 13:42:46.353334 ip-10-0-143-222 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:42:46.354783 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.354696 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 13:42:46.360674 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360655 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:42:46.360674 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360670 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:42:46.360674 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360675 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:42:46.360674 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360680 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360685 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360690 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360697 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360702 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360707 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360712 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360716 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360720 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360725 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360729 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360733 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360737 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360742 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360746 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360750 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360770 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360774 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360778 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:42:46.360916 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360782 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360786 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360790 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360795 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360799 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360803 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360807 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360812 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360816 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360820 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360824 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360829 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360837 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360846 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360853 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360859 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360865 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360869 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360874 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:42:46.361565 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360879 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360884 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360888 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360893 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360897 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360901 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360906 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360910 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360915 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360919 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360923 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360929 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360933 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360937 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360941 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360945 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360950 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360954 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360958 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360962 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:42:46.362023 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360966 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360970 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360974 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360978 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360983 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360988 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360992 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.360998 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361002 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361006 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361011 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361015 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361019 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361023 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361027 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361031 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361036 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361040 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361044 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361048 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:42:46.362595 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361052 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361057 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361063 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361067 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361078 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361082 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361086 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361090 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361094 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361098 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361103 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361107 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361111 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361116 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361120 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361125 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361129 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:42:46.363088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.361134 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:42:46.364377 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364342 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:42:46.364377 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364374 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:42:46.364377 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364380 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364386 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364391 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364397 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364402 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364406 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364413 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364417 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364421 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364426 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364431 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364440 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364444 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364449 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364454 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364458 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364462 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364468 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364472 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364476 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:42:46.364563 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364481 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364486 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364490 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364494 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364502 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364507 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364511 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364515 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364520 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364524 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364529 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364535 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364539 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364543 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364549 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364553 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364562 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364567 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364572 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364577 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:42:46.365226 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364581 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364585 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364589 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364594 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364598 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364602 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364607 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364611 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364616 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364625 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364629 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364639 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364645 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364651 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364656 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364660 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364666 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364670 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364675 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364679 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:42:46.365728 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364683 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364693 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364698 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364702 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364707 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364712 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364717 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364722 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364726 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364730 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364734 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364738 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364743 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364752 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364762 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364769 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364774 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364781 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364787 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:42:46.366258 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364791 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364796 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364801 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364805 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364810 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364819 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364823 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364828 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364832 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364836 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364840 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364845 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364868 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364872 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364877 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364881 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364886 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.364890 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365928 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365939 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:42:46.366735 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365942 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365945 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365948 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365952 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365955 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365958 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365961 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365964 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365967 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365971 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365973 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365976 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365979 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365981 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365984 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365987 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365989 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365992 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365994 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365997 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:42:46.367338 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.365999 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366002 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366004 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366007 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366012 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366016 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366018 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366021 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366024 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366028 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366030 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366033 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366036 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366039 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366041 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366044 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366046 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366049 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366052 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:42:46.367827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366054 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366057 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366059 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366062 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366065 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366067 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366070 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366072 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366076 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366080 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366082 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366085 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366087 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366090 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366092 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366095 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366097 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366099 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366102 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:42:46.368324 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366104 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366107 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366109 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366112 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366115 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366118 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366121 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366124 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366126 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366129 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366131 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366134 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366137 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366153 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366155 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366158 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366160 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366164 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366168 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:42:46.368827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366171 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366174 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366177 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366181 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366184 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366187 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366190 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366193 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366196 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366199 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366201 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366204 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366208 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366211 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366213 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366216 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366219 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366222 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366226 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366228 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366297 2581 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 13:42:46.369316 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366333 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366340 2581 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366344 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366348 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366351 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366355 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366359 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366363 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366366 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366369 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366373 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366376 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366379 2581 flags.go:64] FLAG: --cgroup-root="" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366382 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366385 2581 flags.go:64] FLAG: --client-ca-file="" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366389 2581 flags.go:64] FLAG: --cloud-provider="external" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366392 2581 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366397 2581 flags.go:64] FLAG: --cluster-domain="" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366400 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366403 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366406 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366409 2581 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366418 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 13:42:46.369831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366421 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366424 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366427 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366430 2581 flags.go:64] FLAG: --contention-profiling="false" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366433 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366436 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366440 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366443 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366447 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366450 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366453 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366455 2581 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366458 2581 flags.go:64] FLAG: --enable-server="true" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366461 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366466 2581 flags.go:64] FLAG: --event-burst="100" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366469 2581 flags.go:64] FLAG: --event-qps="50" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366472 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366475 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366477 2581 flags.go:64] FLAG: --eviction-hard="" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366481 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366484 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366487 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366490 2581 flags.go:64] FLAG: --eviction-soft="" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366493 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366495 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 13:42:46.370408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366499 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366502 2581 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366505 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366508 2581 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366511 2581 flags.go:64] FLAG: --feature-gates="" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366515 2581 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366518 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366521 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366524 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366527 2581 flags.go:64] FLAG: --healthz-port="10248" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366530 2581 flags.go:64] FLAG: --help="false" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366533 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366536 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366539 2581 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366542 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366545 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366548 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366551 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366554 2581 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366557 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366560 2581 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366562 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366566 2581 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366568 2581 flags.go:64] FLAG: --kube-reserved="" Sep 16 13:42:46.371046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366571 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366574 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366577 2581 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366579 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366582 2581 flags.go:64] FLAG: --lock-file="" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366585 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366588 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366591 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366596 2581 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366600 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366603 2581 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366606 2581 flags.go:64] FLAG: --logging-format="text" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366608 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366612 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366614 2581 flags.go:64] FLAG: --manifest-url="" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366627 2581 flags.go:64] FLAG: --manifest-url-header="" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366636 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366640 2581 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366644 2581 flags.go:64] FLAG: --max-pods="110" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366647 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366650 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366653 2581 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366656 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366659 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366662 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 13:42:46.371658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366665 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366672 2581 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366675 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366678 2581 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366681 2581 flags.go:64] FLAG: --pod-cidr="" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366684 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366690 2581 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366693 2581 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366696 2581 flags.go:64] FLAG: --pods-per-core="0" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366699 2581 flags.go:64] FLAG: --port="10250" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366702 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366705 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0797dfef7431d1e72" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366708 2581 flags.go:64] FLAG: --qos-reserved="" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366711 2581 flags.go:64] FLAG: --read-only-port="10255" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366714 2581 flags.go:64] FLAG: --register-node="true" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366716 2581 flags.go:64] FLAG: --register-with-taints="" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366720 2581 flags.go:64] FLAG: --registry-burst="10" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366724 2581 flags.go:64] FLAG: --registry-qps="5" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366726 2581 flags.go:64] FLAG: --reserved-cpus="" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366730 2581 flags.go:64] FLAG: --reserved-memory="" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366734 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366737 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366740 2581 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366745 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366748 2581 flags.go:64] FLAG: --runonce="false" Sep 16 13:42:46.372343 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366751 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366755 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366758 2581 flags.go:64] FLAG: --seccomp-default="false" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366761 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366764 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366767 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366770 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366773 2581 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366776 2581 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366779 2581 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366782 2581 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366785 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366788 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366791 2581 flags.go:64] FLAG: --system-cgroups="" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366794 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366799 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366802 2581 flags.go:64] FLAG: --tls-cert-file="" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366805 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366808 2581 flags.go:64] FLAG: --tls-min-version="" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366811 2581 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366814 2581 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366817 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366820 2581 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366823 2581 flags.go:64] FLAG: --v="2" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366826 2581 flags.go:64] FLAG: --version="false" Sep 16 13:42:46.372959 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366833 2581 flags.go:64] FLAG: --vmodule="" Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366837 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.366840 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366956 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366960 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366965 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366968 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366971 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366975 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366978 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366981 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366983 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366986 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366988 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366991 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366994 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.366996 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367000 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367004 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367007 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:42:46.373596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367010 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367013 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367016 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367019 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367021 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367024 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367026 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367029 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367031 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367034 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367036 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367039 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367044 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367047 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367050 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367052 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367055 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367060 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367063 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367066 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:42:46.374134 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367068 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367071 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367073 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367076 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367079 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367081 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367085 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367089 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367091 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367094 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367097 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367100 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367102 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367105 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367108 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367110 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367113 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367115 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367118 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:42:46.374655 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367120 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367123 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367125 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367128 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367131 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367135 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367151 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367155 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367158 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367160 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367165 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367167 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367170 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367172 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367175 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367177 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367180 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367183 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367185 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367188 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:42:46.375153 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367190 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367193 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367195 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367198 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367200 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367203 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367206 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367208 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367211 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367214 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367216 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367219 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367221 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367224 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367226 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367229 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367231 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367235 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367238 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367241 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:42:46.375659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367244 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367247 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.367251 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.367970 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.375418 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.375435 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.375454 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.375460 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375517 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375522 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375526 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375530 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375534 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375538 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:42:46.376167 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375542 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375545 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375547 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375550 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375552 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375555 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375558 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375561 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375563 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375566 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375568 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375571 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375574 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375577 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375579 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375583 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375585 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375588 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375590 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375593 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:42:46.376530 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375596 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375598 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375601 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375606 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375609 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375612 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375614 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375617 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375619 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375622 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375624 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375627 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375630 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375632 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375635 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375638 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375640 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375643 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375645 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375648 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:42:46.377019 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375651 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375653 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375655 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375658 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375661 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375663 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375666 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375669 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375671 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375674 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375677 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375679 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375682 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375684 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375687 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375690 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375693 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375696 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375699 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375701 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:42:46.377583 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375703 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375706 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375709 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375711 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375714 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375718 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375722 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375725 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375727 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375730 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375733 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375736 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375739 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375741 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375744 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375747 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375749 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375752 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375754 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:42:46.378063 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375757 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375759 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375762 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375765 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375768 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375771 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375773 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375776 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375779 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375781 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375785 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375788 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375792 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375795 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:42:46.378534 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.375800 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375905 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375909 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375912 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375915 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375917 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375920 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375923 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375925 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375928 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375932 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375935 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375939 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375941 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375944 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375947 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375949 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375952 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375954 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375957 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:42:46.378914 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375959 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375962 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375965 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375968 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375970 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375973 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375976 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375979 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375982 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375985 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375987 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375990 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375993 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375995 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.375998 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376001 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376003 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376005 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376008 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376011 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:42:46.379423 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376013 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376016 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376019 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376021 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376024 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376027 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376029 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376032 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376034 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376037 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376040 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376042 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376045 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376047 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376050 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376053 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376055 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376058 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376061 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376064 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:42:46.379943 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376066 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376069 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376072 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376075 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376078 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376081 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376083 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376086 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376088 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376091 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376093 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376096 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376098 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376101 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376103 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376106 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376108 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376111 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376113 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:42:46.380480 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376116 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376118 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376121 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376123 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376126 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376128 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376131 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376133 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376136 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376154 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376157 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376160 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376163 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376165 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376167 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376170 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376174 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376178 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376182 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:42:46.380946 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376185 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:42:46.381440 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:46.376188 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:42:46.381440 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.376193 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:42:46.381440 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.376291 2581 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 13:42:46.381440 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.378606 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 13:42:46.381440 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.379735 2581 server.go:1019] "Starting client certificate rotation" Sep 16 13:42:46.381440 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.379837 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 13:42:46.381440 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.379908 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 13:42:46.382567 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.382554 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 13:42:46.384612 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.384596 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 13:42:46.393874 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.393853 2581 log.go:25] "Validated CRI v1 runtime API" Sep 16 13:42:46.397215 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.397202 2581 log.go:25] "Validated CRI v1 image API" Sep 16 13:42:46.398024 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.398004 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 13:42:46.402161 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.402127 2581 fs.go:135] Filesystem UUIDs: map[53882aaf-a045-421a-8ff0-f5b0a8963b2d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b29e9e21-6af5-414e-9e9d-3f6e67ca7533:/dev/nvme0n1p4] Sep 16 13:42:46.402246 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.402160 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 13:42:46.408242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.408090 2581 manager.go:216] Machine: {Timestamp:2026-09-16 13:42:46.405702017 +0000 UTC m=+0.369060836 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2962720 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b55240511076b76d6f77ebba3a797 SystemUUID:ec2b5524-0511-076b-76d6-f77ebba3a797 BootID:bfdbfdd8-a2a2-4588-bf4a-35450c32a3de Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:d3:9a:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:d3:9a:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:a3:67:e8:40:a0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 13:42:46.408242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.408234 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 13:42:46.408401 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.408326 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 13:42:46.411768 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.411744 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 13:42:46.411922 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.411771 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 13:42:46.412725 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.412714 2581 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 13:42:46.412760 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.412727 2581 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 13:42:46.412760 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.412735 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 13:42:46.413571 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.413561 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 13:42:46.414808 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.414798 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 13:42:46.417931 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.417912 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 13:42:46.419345 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.419333 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 16 13:42:46.419448 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.419439 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 13:42:46.419498 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.419490 2581 kubelet.go:486] "Attempting to sync node with API server" Sep 16 13:42:46.419532 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.419518 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 13:42:46.419532 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.419530 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 13:42:46.419586 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.419542 2581 kubelet.go:398] "Adding apiserver pod source" Sep 16 13:42:46.419586 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.419569 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 13:42:46.419917 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.419887 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 13:42:46.419917 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.419904 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 13:42:46.422408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.422385 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 13:42:46.425267 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.425253 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 13:42:46.425330 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.425283 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 13:42:46.427433 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427418 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 13:42:46.427487 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427439 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 13:42:46.427487 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427449 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 13:42:46.427487 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427457 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 13:42:46.427487 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427467 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 13:42:46.427487 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427474 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 13:42:46.427619 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427493 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 13:42:46.427619 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427499 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 13:42:46.427619 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427510 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 13:42:46.427619 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427517 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 13:42:46.427619 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427529 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 13:42:46.427619 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.427536 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 13:42:46.428476 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.428465 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 13:42:46.428476 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.428475 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 13:42:46.430792 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.430778 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 13:42:46.431533 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.431510 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 13:42:46.431576 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.431510 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 13:42:46.432279 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.432176 2581 server.go:1268] "Started kubelet" Sep 16 13:42:46.432356 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.432259 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 13:42:46.432401 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.432297 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 13:42:46.432442 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.432423 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 13:42:46.432998 ip-10-0-143-222 systemd[1]: Started Kubernetes Kubelet. Sep 16 13:42:46.434837 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.434807 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 13:42:46.437718 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.437700 2581 server.go:310] "Adding debug handlers to kubelet server" Sep 16 13:42:46.438587 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.438568 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 13:42:46.444073 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.444048 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 13:42:46.444843 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.444726 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 13:42:46.445187 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.445127 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:46.445859 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.445841 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 13:42:46.445945 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.445848 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 13:42:46.445945 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.445894 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 13:42:46.446311 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.446296 2581 factory.go:55] Registering systemd factory Sep 16 13:42:46.446352 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.446316 2581 factory.go:223] Registration of the systemd container factory successfully Sep 16 13:42:46.446550 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.446532 2581 factory.go:153] Registering CRI-O factory Sep 16 13:42:46.446550 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.446550 2581 factory.go:223] Registration of the crio container factory successfully Sep 16 13:42:46.446711 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.446626 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 13:42:46.446711 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.446645 2581 factory.go:103] Registering Raw factory Sep 16 13:42:46.446711 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.446660 2581 manager.go:1196] Started watching for new ooms in manager Sep 16 13:42:46.447793 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.447745 2581 reconstruct.go:97] "Volume reconstruction finished" Sep 16 13:42:46.447793 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.447758 2581 reconciler.go:29] "Reconciler: start to sync state" Sep 16 13:42:46.449780 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.449763 2581 manager.go:349] Starting recovery of all containers Sep 16 13:42:46.450967 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.450942 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 13:42:46.451058 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.450957 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 13:42:46.452131 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.450989 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-222.ec2.internal.18d5d0f2643e95e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-222.ec2.internal,UID:ip-10-0-143-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-222.ec2.internal,},FirstTimestamp:2026-09-16 13:42:46.432126438 +0000 UTC m=+0.395485255,LastTimestamp:2026-09-16 13:42:46.432126438 +0000 UTC m=+0.395485255,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-222.ec2.internal,}" Sep 16 13:42:46.460704 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.460688 2581 manager.go:354] Recovery completed Sep 16 13:42:46.464781 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.464768 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:42:46.466595 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.466570 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:42:46.466695 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.466613 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:42:46.466695 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.466636 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:42:46.467233 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.467219 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 13:42:46.467233 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.467232 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 13:42:46.467366 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.467251 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 16 13:42:46.467366 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.467277 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:42:46.467989 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.467972 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:42:46.468064 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.468004 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:42:46.468064 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.468020 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:42:46.469263 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.469186 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-222.ec2.internal.18d5d0f2664c89f2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-222.ec2.internal,UID:ip-10-0-143-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-222.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-222.ec2.internal,},FirstTimestamp:2026-09-16 13:42:46.466595314 +0000 UTC m=+0.429954136,LastTimestamp:2026-09-16 13:42:46.466595314 +0000 UTC m=+0.429954136,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-222.ec2.internal,}" Sep 16 13:42:46.469464 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.469449 2581 policy_none.go:49] "None policy: Start" Sep 16 13:42:46.469535 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.469468 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 13:42:46.469535 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.469482 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 13:42:46.471480 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.471464 2581 policy_none.go:47] "Start" Sep 16 13:42:46.477818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.477799 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sgsdj" Sep 16 13:42:46.479894 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.479833 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-222.ec2.internal.18d5d0f2664d0737 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-222.ec2.internal,UID:ip-10-0-143-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-222.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-222.ec2.internal,},FirstTimestamp:2026-09-16 13:42:46.466627383 +0000 UTC m=+0.429986202,LastTimestamp:2026-09-16 13:42:46.466627383 +0000 UTC m=+0.429986202,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-222.ec2.internal,}" Sep 16 13:42:46.488199 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.488181 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sgsdj" Sep 16 13:42:46.489378 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.489320 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-222.ec2.internal.18d5d0f2664d42ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-222.ec2.internal,UID:ip-10-0-143-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-222.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-222.ec2.internal,},FirstTimestamp:2026-09-16 13:42:46.466642604 +0000 UTC m=+0.430001424,LastTimestamp:2026-09-16 13:42:46.466642604 +0000 UTC m=+0.430001424,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-222.ec2.internal,}" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.507524 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.508660 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.508712 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.508737 2581 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.508779 2581 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.510835 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.510839 2581 manager.go:337] "Starting Device Plugin manager" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.510978 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.510992 2581 server.go:87] "Starting device plugin registration server" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.511339 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.511349 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.511481 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.511574 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.511587 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.511993 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 13:42:46.518489 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.512045 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:46.609295 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.609171 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal"] Sep 16 13:42:46.609295 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.609288 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:42:46.610749 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.610729 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:42:46.610825 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.610765 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:42:46.610825 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.610778 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:42:46.612434 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.611420 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:42:46.613559 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.613540 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:42:46.613677 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.613636 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.613677 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.613669 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:42:46.613936 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.613919 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:42:46.613997 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.613954 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:42:46.613997 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.613966 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:42:46.613997 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.613988 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.614336 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.614317 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:42:46.614432 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.614349 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:42:46.614432 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.614368 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:42:46.614432 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.614379 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:42:46.614432 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.614352 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:42:46.614554 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.614441 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:42:46.616765 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.616748 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.616846 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.616772 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:42:46.617526 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.617514 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:42:46.617585 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.617532 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:42:46.617585 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.617542 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:42:46.629325 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.629309 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.629400 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.629331 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-222.ec2.internal\": node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:46.648416 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.648386 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ff7c77e7f1b706d9871c56b53d5c67e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal\" (UID: \"9ff7c77e7f1b706d9871c56b53d5c67e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.648416 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.648417 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ff7c77e7f1b706d9871c56b53d5c67e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal\" (UID: \"9ff7c77e7f1b706d9871c56b53d5c67e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.648629 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.648434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/039e38a7f72228dfa18bca5c2b7c263f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-222.ec2.internal\" (UID: \"039e38a7f72228dfa18bca5c2b7c263f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.651113 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.651095 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-222.ec2.internal\" not found" node="ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.654717 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.654693 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-222.ec2.internal\" not found" node="ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.661036 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.661019 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:46.749075 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.749045 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ff7c77e7f1b706d9871c56b53d5c67e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal\" (UID: \"9ff7c77e7f1b706d9871c56b53d5c67e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.749075 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.749076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/039e38a7f72228dfa18bca5c2b7c263f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-222.ec2.internal\" (UID: \"039e38a7f72228dfa18bca5c2b7c263f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.749272 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.749090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ff7c77e7f1b706d9871c56b53d5c67e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal\" (UID: \"9ff7c77e7f1b706d9871c56b53d5c67e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.749272 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.749135 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ff7c77e7f1b706d9871c56b53d5c67e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal\" (UID: \"9ff7c77e7f1b706d9871c56b53d5c67e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.749272 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.749172 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/039e38a7f72228dfa18bca5c2b7c263f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-222.ec2.internal\" (UID: \"039e38a7f72228dfa18bca5c2b7c263f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.749272 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.749174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ff7c77e7f1b706d9871c56b53d5c67e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal\" (UID: \"9ff7c77e7f1b706d9871c56b53d5c67e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.762036 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.762013 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:46.863179 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.863088 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:46.954948 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.954918 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.958598 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:46.958582 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" Sep 16 13:42:46.963968 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:46.963950 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:47.064428 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:47.064397 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:47.164983 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:47.164920 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:47.265416 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:47.265388 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:47.350808 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.350787 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:42:47.366012 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:47.365988 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:47.378906 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.378882 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 13:42:47.379016 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.378999 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 13:42:47.379067 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.379047 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 13:42:47.466338 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:47.466273 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:47.494168 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.494119 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 13:37:46 +0000 UTC" deadline="2028-05-05 03:50:34.999708006 +0000 UTC" Sep 16 13:42:47.494168 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.494163 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14318h7m47.505548369s" Sep 16 13:42:47.556229 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.556196 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:42:47.567353 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:47.567333 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-222.ec2.internal\" not found" Sep 16 13:42:47.618279 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.618257 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:42:47.646251 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.646229 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal" Sep 16 13:42:47.662307 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.662286 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 13:42:47.663658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.663638 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" Sep 16 13:42:47.689071 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.689041 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 13:42:47.752954 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:47.752918 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039e38a7f72228dfa18bca5c2b7c263f.slice/crio-07a326e2d6579cd3f56bdebef8e19ade0be7584467b4a44834ad1df501aad9fa WatchSource:0}: Error finding container 07a326e2d6579cd3f56bdebef8e19ade0be7584467b4a44834ad1df501aad9fa: Status 404 returned error can't find the container with id 07a326e2d6579cd3f56bdebef8e19ade0be7584467b4a44834ad1df501aad9fa Sep 16 13:42:47.758969 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:47.758949 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff7c77e7f1b706d9871c56b53d5c67e.slice/crio-f5ca7cba894a777ba48337c76389347f9ea1943395fc8071eb9206834788a702 WatchSource:0}: Error finding container f5ca7cba894a777ba48337c76389347f9ea1943395fc8071eb9206834788a702: Status 404 returned error can't find the container with id f5ca7cba894a777ba48337c76389347f9ea1943395fc8071eb9206834788a702 Sep 16 13:42:47.760787 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:47.760773 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:42:48.422129 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.422101 2581 apiserver.go:52] "Watching apiserver" Sep 16 13:42:48.429005 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.428985 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 13:42:48.431544 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.431522 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-zmd5t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5","openshift-image-registry/node-ca-n97z8","openshift-multus/multus-kzjdt","openshift-multus/network-metrics-daemon-9dzbs","openshift-network-diagnostics/network-check-target-mcl5r","openshift-ovn-kubernetes/ovnkube-node-g58wg","kube-system/global-pull-secret-syncer-hddml","kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal","openshift-cluster-node-tuning-operator/tuned-hnw8s","openshift-dns/node-resolver-gvnzs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal","openshift-multus/multus-additional-cni-plugins-4pwdh","openshift-network-operator/iptables-alerter-vt5fx"] Sep 16 13:42:48.435401 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.435383 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.437768 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.437747 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.438297 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.438270 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 13:42:48.438397 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.438281 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 13:42:48.438822 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.438787 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 13:42:48.439031 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.439008 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvl5l\"" Sep 16 13:42:48.439253 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.439162 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 13:42:48.439363 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.439259 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 13:42:48.439557 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.439538 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 13:42:48.441105 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.440793 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 13:42:48.441105 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.441054 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 13:42:48.442376 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.442356 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.443085 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.442636 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 13:42:48.443085 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.442912 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ddzvl\"" Sep 16 13:42:48.443085 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.442949 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 13:42:48.443085 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.443073 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 13:42:48.444780 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.444762 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.444879 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.444789 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 13:42:48.445759 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.445738 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 13:42:48.445875 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.445765 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 13:42:48.447231 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.447194 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 13:42:48.447231 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.447213 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:48.447381 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.447284 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:42:48.447992 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.447975 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kj99r\"" Sep 16 13:42:48.448078 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.448055 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 13:42:48.448410 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.448395 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7nvjh\"" Sep 16 13:42:48.449742 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.449685 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:48.449818 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.449746 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:42:48.452101 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.452084 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.454489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.454470 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:42:48.454841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.454826 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 13:42:48.455190 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.455175 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:42:48.455414 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.455402 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 13:42:48.456148 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.456127 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xdhdr\"" Sep 16 13:42:48.458101 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.457631 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 13:42:48.458101 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.457652 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 13:42:48.458101 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.457640 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.458101 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.457969 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-958k5\"" Sep 16 13:42:48.458335 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.458296 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 13:42:48.459767 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.459746 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23cd4ca2-227a-4011-ba43-5afafff1d0b9-env-overrides\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.459848 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.459782 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjtzd\" (UniqueName: \"kubernetes.io/projected/f4305a9e-2784-4ecd-9462-64b45a220538-kube-api-access-qjtzd\") pod \"node-ca-n97z8\" (UID: \"f4305a9e-2784-4ecd-9462-64b45a220538\") " pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.459848 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.459809 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-hostroot\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.459848 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.459824 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-run-systemd\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.459848 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.459842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23cd4ca2-227a-4011-ba43-5afafff1d0b9-ovn-node-metrics-cert\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.460045 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.459951 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75v2b\" (UniqueName: \"kubernetes.io/projected/23cd4ca2-227a-4011-ba43-5afafff1d0b9-kube-api-access-75v2b\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.460045 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.459982 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/740f47ff-eb88-4272-8677-19e1d3c2148f-system-cni-dir\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.460045 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-run-k8s-cni-cncf-io\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.460045 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460034 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-cni-dir\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.460282 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-socket-dir-parent\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.460282 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-systemd-units\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.460282 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460122 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.460282 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460167 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-etc-openvswitch\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.460282 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460219 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.460282 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/740f47ff-eb88-4272-8677-19e1d3c2148f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.460583 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460309 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-run-netns\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.460583 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-var-lib-kubelet\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.460583 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460404 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-var-lib-openvswitch\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.460583 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460439 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-run-openvswitch\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.460583 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.460583 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460507 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/740f47ff-eb88-4272-8677-19e1d3c2148f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.460583 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460556 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhmq2\" (UniqueName: \"kubernetes.io/projected/740f47ff-eb88-4272-8677-19e1d3c2148f-kube-api-access-vhmq2\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.460930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4305a9e-2784-4ecd-9462-64b45a220538-host\") pod \"node-ca-n97z8\" (UID: \"f4305a9e-2784-4ecd-9462-64b45a220538\") " pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.460930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4305a9e-2784-4ecd-9462-64b45a220538-serviceca\") pod \"node-ca-n97z8\" (UID: \"f4305a9e-2784-4ecd-9462-64b45a220538\") " pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.460930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebd160ee-64b4-4ca0-9368-510cb8bab449-cni-binary-copy\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.460930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460688 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-daemon-config\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.460930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltp2x\" (UniqueName: \"kubernetes.io/projected/ebd160ee-64b4-4ca0-9368-510cb8bab449-kube-api-access-ltp2x\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.460930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-var-lib-cni-multus\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.460930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-conf-dir\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.461288 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460957 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 13:42:48.461288 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.460974 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 13:42:48.461288 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqqv\" (UniqueName: \"kubernetes.io/projected/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-kube-api-access-4sqqv\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:48.461472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461351 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-kubelet\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.461472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461376 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-slash\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.461472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461393 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23cd4ca2-227a-4011-ba43-5afafff1d0b9-ovnkube-script-lib\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.461472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461396 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 13:42:48.461472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461415 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/740f47ff-eb88-4272-8677-19e1d3c2148f-cnibin\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.461472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461441 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-run-multus-certs\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.461472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461469 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-node-log\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461496 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-cni-bin\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461522 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/740f47ff-eb88-4272-8677-19e1d3c2148f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461545 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-cnibin\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461579 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-var-lib-cni-bin\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461605 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461632 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-cni-netd\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461660 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23cd4ca2-227a-4011-ba43-5afafff1d0b9-ovnkube-config\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461686 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/740f47ff-eb88-4272-8677-19e1d3c2148f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461711 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-run-netns\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461729 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-system-cni-dir\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461748 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nslqn\"" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461765 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-etc-kubernetes\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-run-ovn\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.461841 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.461818 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-log-socket\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.463023 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.463002 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.463231 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.463048 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2b57\"" Sep 16 13:42:48.463307 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.463292 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 13:42:48.463574 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.463556 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:42:48.465737 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.465715 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:48.465838 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.465773 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:42:48.465838 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.465830 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 13:42:48.466279 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.466256 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 13:42:48.466807 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.466792 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mrs6j\"" Sep 16 13:42:48.488085 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.488063 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgn4v" Sep 16 13:42:48.497982 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.497961 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgn4v" Sep 16 13:42:48.515200 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.515141 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal" event={"ID":"039e38a7f72228dfa18bca5c2b7c263f","Type":"ContainerStarted","Data":"07a326e2d6579cd3f56bdebef8e19ade0be7584467b4a44834ad1df501aad9fa"} Sep 16 13:42:48.516462 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.516439 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" event={"ID":"9ff7c77e7f1b706d9871c56b53d5c67e","Type":"ContainerStarted","Data":"f5ca7cba894a777ba48337c76389347f9ea1943395fc8071eb9206834788a702"} Sep 16 13:42:48.546695 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.546671 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 13:42:48.562526 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebd160ee-64b4-4ca0-9368-510cb8bab449-cni-binary-copy\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.562630 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562534 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltp2x\" (UniqueName: \"kubernetes.io/projected/ebd160ee-64b4-4ca0-9368-510cb8bab449-kube-api-access-ltp2x\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.562630 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562553 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1543153d-e6e7-44f0-881e-9f78070b70c7-agent-certs\") pod \"konnectivity-agent-zmd5t\" (UID: \"1543153d-e6e7-44f0-881e-9f78070b70c7\") " pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:42:48.562630 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562570 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-conf-dir\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.562630 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562591 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-kubelet\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.562630 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562615 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-slash\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.562867 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562744 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-slash\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.562867 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562758 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-conf-dir\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.562867 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562790 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-kubelet\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.562867 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-sysconfig\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.562867 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1499dfa1-b0ce-44a0-add8-c1404440ecdd-kubelet-config\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:48.563158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562878 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-node-log\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562927 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-node-log\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562923 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-cni-bin\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562970 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-var-lib-cni-bin\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.563158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562977 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-cni-bin\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.562987 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:48.563158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23cd4ca2-227a-4011-ba43-5afafff1d0b9-ovnkube-config\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563022 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-var-lib-cni-bin\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.563158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-run-netns\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563178 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-system-cni-dir\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.563130 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-etc-kubernetes\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563203 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-run-netns\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-log-socket\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563249 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-etc-kubernetes\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563273 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23cd4ca2-227a-4011-ba43-5afafff1d0b9-env-overrides\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.563300 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs podName:42514386-d0ca-4a5b-b5aa-d0e5b3276cd0 nodeName:}" failed. No retries permitted until 2026-09-16 13:42:49.06327391 +0000 UTC m=+3.026632736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs") pod "network-metrics-daemon-9dzbs" (UID: "42514386-d0ca-4a5b-b5aa-d0e5b3276cd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563322 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebd160ee-64b4-4ca0-9368-510cb8bab449-cni-binary-copy\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23cd4ca2-227a-4011-ba43-5afafff1d0b9-ovn-node-metrics-cert\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75v2b\" (UniqueName: \"kubernetes.io/projected/23cd4ca2-227a-4011-ba43-5afafff1d0b9-kube-api-access-75v2b\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563366 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-system-cni-dir\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563393 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtzd\" (UniqueName: \"kubernetes.io/projected/f4305a9e-2784-4ecd-9462-64b45a220538-kube-api-access-qjtzd\") pod \"node-ca-n97z8\" (UID: \"f4305a9e-2784-4ecd-9462-64b45a220538\") " pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-hostroot\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563439 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-log-socket\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563473 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/740f47ff-eb88-4272-8677-19e1d3c2148f-system-cni-dir\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.563616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563503 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-sys-fs\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563551 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1543153d-e6e7-44f0-881e-9f78070b70c7-konnectivity-ca\") pod \"konnectivity-agent-zmd5t\" (UID: \"1543153d-e6e7-44f0-881e-9f78070b70c7\") " pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563558 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-hostroot\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563562 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23cd4ca2-227a-4011-ba43-5afafff1d0b9-ovnkube-config\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563620 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1499dfa1-b0ce-44a0-add8-c1404440ecdd-dbus\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563648 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-socket-dir-parent\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/740f47ff-eb88-4272-8677-19e1d3c2148f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563711 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-device-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563726 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/740f47ff-eb88-4272-8677-19e1d3c2148f-system-cni-dir\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563781 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-modprobe-d\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563779 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-socket-dir-parent\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563809 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-host\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563833 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdsn6\" (UniqueName: \"kubernetes.io/projected/7c27e30f-2283-4a75-bffd-fe46cda06f05-kube-api-access-rdsn6\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563859 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696t4\" (UniqueName: \"kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4\") pod \"network-check-target-mcl5r\" (UID: \"f0d7ecc6-7b55-4cca-9c55-093d59c1e854\") " pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563898 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-run-netns\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-var-lib-kubelet\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564107 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-var-lib-openvswitch\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.564492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564124 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564143 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwsqx\" (UniqueName: \"kubernetes.io/projected/46165266-25c4-426d-9917-f0937f9df64a-kube-api-access-wwsqx\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4305a9e-2784-4ecd-9462-64b45a220538-host\") pod \"node-ca-n97z8\" (UID: \"f4305a9e-2784-4ecd-9462-64b45a220538\") " pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4305a9e-2784-4ecd-9462-64b45a220538-serviceca\") pod \"node-ca-n97z8\" (UID: \"f4305a9e-2784-4ecd-9462-64b45a220538\") " pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564210 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-daemon-config\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564207 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/740f47ff-eb88-4272-8677-19e1d3c2148f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.563960 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-run-netns\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564056 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564230 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd6779bd-b278-4b06-a7ca-80a4aae73ae0-hosts-file\") pod \"node-resolver-gvnzs\" (UID: \"cd6779bd-b278-4b06-a7ca-80a4aae73ae0\") " pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564296 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-var-lib-openvswitch\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564298 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4305a9e-2784-4ecd-9462-64b45a220538-host\") pod \"node-ca-n97z8\" (UID: \"f4305a9e-2784-4ecd-9462-64b45a220538\") " pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564297 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-var-lib-cni-multus\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564337 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-var-lib-cni-multus\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564335 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqqv\" (UniqueName: \"kubernetes.io/projected/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-kube-api-access-4sqqv\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23cd4ca2-227a-4011-ba43-5afafff1d0b9-ovnkube-script-lib\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564409 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/740f47ff-eb88-4272-8677-19e1d3c2148f-cnibin\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564437 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/740f47ff-eb88-4272-8677-19e1d3c2148f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.565283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564449 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564494 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-sysctl-d\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564385 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23cd4ca2-227a-4011-ba43-5afafff1d0b9-env-overrides\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564521 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-systemd\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564006 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-var-lib-kubelet\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564569 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-run-multus-certs\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564617 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/740f47ff-eb88-4272-8677-19e1d3c2148f-cnibin\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564704 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-sysctl-conf\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564735 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c27e30f-2283-4a75-bffd-fe46cda06f05-tmp\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564746 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4305a9e-2784-4ecd-9462-64b45a220538-serviceca\") pod \"node-ca-n97z8\" (UID: \"f4305a9e-2784-4ecd-9462-64b45a220538\") " pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564782 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-run-multus-certs\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564779 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/740f47ff-eb88-4272-8677-19e1d3c2148f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564816 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97fc753e-b824-4150-b7e1-051cdd2992ed-iptables-alerter-script\") pod \"iptables-alerter-vt5fx\" (UID: \"97fc753e-b824-4150-b7e1-051cdd2992ed\") " pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwxnd\" (UniqueName: \"kubernetes.io/projected/97fc753e-b824-4150-b7e1-051cdd2992ed-kube-api-access-dwxnd\") pod \"iptables-alerter-vt5fx\" (UID: \"97fc753e-b824-4150-b7e1-051cdd2992ed\") " pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564878 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-cnibin\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564899 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-daemon-config\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.565974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-cni-netd\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564946 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-cni-netd\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.564958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/740f47ff-eb88-4272-8677-19e1d3c2148f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565002 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-kubernetes\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565006 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-cnibin\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565037 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-run-ovn\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-socket-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lmpt\" (UniqueName: \"kubernetes.io/projected/cd6779bd-b278-4b06-a7ca-80a4aae73ae0-kube-api-access-4lmpt\") pod \"node-resolver-gvnzs\" (UID: \"cd6779bd-b278-4b06-a7ca-80a4aae73ae0\") " pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565154 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-run-ovn\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565173 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-sys\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-lib-modules\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-run-systemd\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-registration-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565320 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23cd4ca2-227a-4011-ba43-5afafff1d0b9-ovnkube-script-lib\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565320 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-var-lib-kubelet\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565356 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-run-systemd\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-tuned\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.566857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565392 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565415 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97fc753e-b824-4150-b7e1-051cdd2992ed-host-slash\") pod \"iptables-alerter-vt5fx\" (UID: \"97fc753e-b824-4150-b7e1-051cdd2992ed\") " pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-run-k8s-cni-cncf-io\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565463 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-cni-dir\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565480 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-systemd-units\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565531 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/740f47ff-eb88-4272-8677-19e1d3c2148f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-etc-openvswitch\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565537 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-host-run-k8s-cni-cncf-io\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565575 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-etc-openvswitch\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565584 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565607 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-systemd-units\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565616 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565644 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6779bd-b278-4b06-a7ca-80a4aae73ae0-tmp-dir\") pod \"node-resolver-gvnzs\" (UID: \"cd6779bd-b278-4b06-a7ca-80a4aae73ae0\") " pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565633 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebd160ee-64b4-4ca0-9368-510cb8bab449-multus-cni-dir\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565650 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565682 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-run\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-run-openvswitch\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.567698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/740f47ff-eb88-4272-8677-19e1d3c2148f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.568565 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565766 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmq2\" (UniqueName: \"kubernetes.io/projected/740f47ff-eb88-4272-8677-19e1d3c2148f-kube-api-access-vhmq2\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.568565 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.565785 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23cd4ca2-227a-4011-ba43-5afafff1d0b9-run-openvswitch\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.568565 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.566164 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/740f47ff-eb88-4272-8677-19e1d3c2148f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.569099 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.569076 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23cd4ca2-227a-4011-ba43-5afafff1d0b9-ovn-node-metrics-cert\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.574153 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.574130 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtzd\" (UniqueName: \"kubernetes.io/projected/f4305a9e-2784-4ecd-9462-64b45a220538-kube-api-access-qjtzd\") pod \"node-ca-n97z8\" (UID: \"f4305a9e-2784-4ecd-9462-64b45a220538\") " pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.574302 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.574176 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltp2x\" (UniqueName: \"kubernetes.io/projected/ebd160ee-64b4-4ca0-9368-510cb8bab449-kube-api-access-ltp2x\") pod \"multus-kzjdt\" (UID: \"ebd160ee-64b4-4ca0-9368-510cb8bab449\") " pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.574970 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.574943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqqv\" (UniqueName: \"kubernetes.io/projected/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-kube-api-access-4sqqv\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:48.575132 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.575113 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75v2b\" (UniqueName: \"kubernetes.io/projected/23cd4ca2-227a-4011-ba43-5afafff1d0b9-kube-api-access-75v2b\") pod \"ovnkube-node-g58wg\" (UID: \"23cd4ca2-227a-4011-ba43-5afafff1d0b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.577560 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.577535 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmq2\" (UniqueName: \"kubernetes.io/projected/740f47ff-eb88-4272-8677-19e1d3c2148f-kube-api-access-vhmq2\") pod \"multus-additional-cni-plugins-4pwdh\" (UID: \"740f47ff-eb88-4272-8677-19e1d3c2148f\") " pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.666044 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666008 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-sysconfig\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.666260 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1499dfa1-b0ce-44a0-add8-c1404440ecdd-kubelet-config\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:48.666260 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-sys-fs\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.666260 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666186 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1543153d-e6e7-44f0-881e-9f78070b70c7-konnectivity-ca\") pod \"konnectivity-agent-zmd5t\" (UID: \"1543153d-e6e7-44f0-881e-9f78070b70c7\") " pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:42:48.666260 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666206 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-sysconfig\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.666260 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666210 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1499dfa1-b0ce-44a0-add8-c1404440ecdd-dbus\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:48.666260 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666239 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-device-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.666260 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666262 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-modprobe-d\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666278 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-sys-fs\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-host\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666317 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1499dfa1-b0ce-44a0-add8-c1404440ecdd-dbus\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666148 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1499dfa1-b0ce-44a0-add8-c1404440ecdd-kubelet-config\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666339 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-device-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666373 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-host\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666388 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdsn6\" (UniqueName: \"kubernetes.io/projected/7c27e30f-2283-4a75-bffd-fe46cda06f05-kube-api-access-rdsn6\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666417 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696t4\" (UniqueName: \"kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4\") pod \"network-check-target-mcl5r\" (UID: \"f0d7ecc6-7b55-4cca-9c55-093d59c1e854\") " pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666426 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-modprobe-d\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666446 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsqx\" (UniqueName: \"kubernetes.io/projected/46165266-25c4-426d-9917-f0937f9df64a-kube-api-access-wwsqx\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd6779bd-b278-4b06-a7ca-80a4aae73ae0-hosts-file\") pod \"node-resolver-gvnzs\" (UID: \"cd6779bd-b278-4b06-a7ca-80a4aae73ae0\") " pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.666616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666584 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd6779bd-b278-4b06-a7ca-80a4aae73ae0-hosts-file\") pod \"node-resolver-gvnzs\" (UID: \"cd6779bd-b278-4b06-a7ca-80a4aae73ae0\") " pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666635 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-sysctl-d\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-systemd\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666735 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-sysctl-conf\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c27e30f-2283-4a75-bffd-fe46cda06f05-tmp\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97fc753e-b824-4150-b7e1-051cdd2992ed-iptables-alerter-script\") pod \"iptables-alerter-vt5fx\" (UID: \"97fc753e-b824-4150-b7e1-051cdd2992ed\") " pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666813 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwxnd\" (UniqueName: \"kubernetes.io/projected/97fc753e-b824-4150-b7e1-051cdd2992ed-kube-api-access-dwxnd\") pod \"iptables-alerter-vt5fx\" (UID: \"97fc753e-b824-4150-b7e1-051cdd2992ed\") " pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666844 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-kubernetes\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666856 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-sysctl-d\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666888 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-systemd\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666898 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-socket-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666905 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-sysctl-conf\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666924 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1543153d-e6e7-44f0-881e-9f78070b70c7-konnectivity-ca\") pod \"konnectivity-agent-zmd5t\" (UID: \"1543153d-e6e7-44f0-881e-9f78070b70c7\") " pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lmpt\" (UniqueName: \"kubernetes.io/projected/cd6779bd-b278-4b06-a7ca-80a4aae73ae0-kube-api-access-4lmpt\") pod \"node-resolver-gvnzs\" (UID: \"cd6779bd-b278-4b06-a7ca-80a4aae73ae0\") " pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-sys\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666989 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-kubernetes\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.667294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667036 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-sys\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.666991 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-lib-modules\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667081 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-lib-modules\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-registration-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667123 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-var-lib-kubelet\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667129 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-registration-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-tuned\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667140 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-socket-dir\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-var-lib-kubelet\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667228 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97fc753e-b824-4150-b7e1-051cdd2992ed-host-slash\") pod \"iptables-alerter-vt5fx\" (UID: \"97fc753e-b824-4150-b7e1-051cdd2992ed\") " pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667253 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.667263 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.667323 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret podName:1499dfa1-b0ce-44a0-add8-c1404440ecdd nodeName:}" failed. No retries permitted until 2026-09-16 13:42:49.167304649 +0000 UTC m=+3.130663467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret") pod "global-pull-secret-syncer-hddml" (UID: "1499dfa1-b0ce-44a0-add8-c1404440ecdd") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6779bd-b278-4b06-a7ca-80a4aae73ae0-tmp-dir\") pod \"node-resolver-gvnzs\" (UID: \"cd6779bd-b278-4b06-a7ca-80a4aae73ae0\") " pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667363 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-run\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1543153d-e6e7-44f0-881e-9f78070b70c7-agent-certs\") pod \"konnectivity-agent-zmd5t\" (UID: \"1543153d-e6e7-44f0-881e-9f78070b70c7\") " pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:42:48.668110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667490 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97fc753e-b824-4150-b7e1-051cdd2992ed-iptables-alerter-script\") pod \"iptables-alerter-vt5fx\" (UID: \"97fc753e-b824-4150-b7e1-051cdd2992ed\") " pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.668980 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667508 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6779bd-b278-4b06-a7ca-80a4aae73ae0-tmp-dir\") pod \"node-resolver-gvnzs\" (UID: \"cd6779bd-b278-4b06-a7ca-80a4aae73ae0\") " pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.668980 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667524 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c27e30f-2283-4a75-bffd-fe46cda06f05-run\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.668980 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667564 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97fc753e-b824-4150-b7e1-051cdd2992ed-host-slash\") pod \"iptables-alerter-vt5fx\" (UID: \"97fc753e-b824-4150-b7e1-051cdd2992ed\") " pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.668980 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.667612 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46165266-25c4-426d-9917-f0937f9df64a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.669773 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.669748 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c27e30f-2283-4a75-bffd-fe46cda06f05-tmp\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.670010 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.669983 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c27e30f-2283-4a75-bffd-fe46cda06f05-etc-tuned\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.670273 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.670249 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1543153d-e6e7-44f0-881e-9f78070b70c7-agent-certs\") pod \"konnectivity-agent-zmd5t\" (UID: \"1543153d-e6e7-44f0-881e-9f78070b70c7\") " pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:42:48.679928 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.679865 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:42:48.679928 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.679896 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:42:48.679928 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.679911 2581 projected.go:196] Error preparing data for projected volume kube-api-access-696t4 for pod openshift-network-diagnostics/network-check-target-mcl5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:42:48.680153 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:48.680030 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4 podName:f0d7ecc6-7b55-4cca-9c55-093d59c1e854 nodeName:}" failed. No retries permitted until 2026-09-16 13:42:49.180001897 +0000 UTC m=+3.143360725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-696t4" (UniqueName: "kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4") pod "network-check-target-mcl5r" (UID: "f0d7ecc6-7b55-4cca-9c55-093d59c1e854") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:42:48.681107 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.681084 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsqx\" (UniqueName: \"kubernetes.io/projected/46165266-25c4-426d-9917-f0937f9df64a-kube-api-access-wwsqx\") pod \"aws-ebs-csi-driver-node-9j9l5\" (UID: \"46165266-25c4-426d-9917-f0937f9df64a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.683556 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.683529 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdsn6\" (UniqueName: \"kubernetes.io/projected/7c27e30f-2283-4a75-bffd-fe46cda06f05-kube-api-access-rdsn6\") pod \"tuned-hnw8s\" (UID: \"7c27e30f-2283-4a75-bffd-fe46cda06f05\") " pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.683672 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.683651 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwxnd\" (UniqueName: \"kubernetes.io/projected/97fc753e-b824-4150-b7e1-051cdd2992ed-kube-api-access-dwxnd\") pod \"iptables-alerter-vt5fx\" (UID: \"97fc753e-b824-4150-b7e1-051cdd2992ed\") " pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.685488 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.685465 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lmpt\" (UniqueName: \"kubernetes.io/projected/cd6779bd-b278-4b06-a7ca-80a4aae73ae0-kube-api-access-4lmpt\") pod \"node-resolver-gvnzs\" (UID: \"cd6779bd-b278-4b06-a7ca-80a4aae73ae0\") " pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.746524 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.746486 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:42:48.752933 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.752902 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" Sep 16 13:42:48.758806 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:48.758779 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cd4ca2_227a_4011_ba43_5afafff1d0b9.slice/crio-a49d9382a76242dc0ce3800090ba42aa9e5abda869be0fb103661df4fe5f63b7 WatchSource:0}: Error finding container a49d9382a76242dc0ce3800090ba42aa9e5abda869be0fb103661df4fe5f63b7: Status 404 returned error can't find the container with id a49d9382a76242dc0ce3800090ba42aa9e5abda869be0fb103661df4fe5f63b7 Sep 16 13:42:48.762823 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.762796 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n97z8" Sep 16 13:42:48.764269 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:48.764234 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740f47ff_eb88_4272_8677_19e1d3c2148f.slice/crio-d6388d9798ce6896592be9d290256e5f7063f5f480d75de73f835dd8e0d97882 WatchSource:0}: Error finding container d6388d9798ce6896592be9d290256e5f7063f5f480d75de73f835dd8e0d97882: Status 404 returned error can't find the container with id d6388d9798ce6896592be9d290256e5f7063f5f480d75de73f835dd8e0d97882 Sep 16 13:42:48.768015 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.767992 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kzjdt" Sep 16 13:42:48.775704 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:48.775676 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4305a9e_2784_4ecd_9462_64b45a220538.slice/crio-3421ed94d07a6c7d550380924bbc6eefab0792084d028e05a3bcdf52c944bcbf WatchSource:0}: Error finding container 3421ed94d07a6c7d550380924bbc6eefab0792084d028e05a3bcdf52c944bcbf: Status 404 returned error can't find the container with id 3421ed94d07a6c7d550380924bbc6eefab0792084d028e05a3bcdf52c944bcbf Sep 16 13:42:48.777930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.777854 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vt5fx" Sep 16 13:42:48.779300 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:48.779053 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd160ee_64b4_4ca0_9368_510cb8bab449.slice/crio-2d1b4a2890b035341f95c461315192615ab6fbbdcf5609c4c13c97b11543c9e0 WatchSource:0}: Error finding container 2d1b4a2890b035341f95c461315192615ab6fbbdcf5609c4c13c97b11543c9e0: Status 404 returned error can't find the container with id 2d1b4a2890b035341f95c461315192615ab6fbbdcf5609c4c13c97b11543c9e0 Sep 16 13:42:48.784579 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.784411 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:42:48.789853 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:48.789826 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fc753e_b824_4150_b7e1_051cdd2992ed.slice/crio-c380b3e5e150f3a191a421e44ec14bcd495e7252cade560ba95743c0aca7d557 WatchSource:0}: Error finding container c380b3e5e150f3a191a421e44ec14bcd495e7252cade560ba95743c0aca7d557: Status 404 returned error can't find the container with id c380b3e5e150f3a191a421e44ec14bcd495e7252cade560ba95743c0aca7d557 Sep 16 13:42:48.791073 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.790998 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" Sep 16 13:42:48.798663 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.798643 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" Sep 16 13:42:48.803549 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:48.803257 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46165266_25c4_426d_9917_f0937f9df64a.slice/crio-3211e15fd5476de160a5c21cc8b6964e408454b8258a46c3989ceb15c5b31749 WatchSource:0}: Error finding container 3211e15fd5476de160a5c21cc8b6964e408454b8258a46c3989ceb15c5b31749: Status 404 returned error can't find the container with id 3211e15fd5476de160a5c21cc8b6964e408454b8258a46c3989ceb15c5b31749 Sep 16 13:42:48.803982 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.803872 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gvnzs" Sep 16 13:42:48.817088 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:48.817063 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c27e30f_2283_4a75_bffd_fe46cda06f05.slice/crio-6500d35389e18a980da9f4ebd0178ff279a279f58e9c2a538c8de79a1f99aa33 WatchSource:0}: Error finding container 6500d35389e18a980da9f4ebd0178ff279a279f58e9c2a538c8de79a1f99aa33: Status 404 returned error can't find the container with id 6500d35389e18a980da9f4ebd0178ff279a279f58e9c2a538c8de79a1f99aa33 Sep 16 13:42:48.822047 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:42:48.822007 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6779bd_b278_4b06_a7ca_80a4aae73ae0.slice/crio-9bdfe3c8e44691b106ce3dc55115577c81f08f1ea7713f0d2e793d6516c71058 WatchSource:0}: Error finding container 9bdfe3c8e44691b106ce3dc55115577c81f08f1ea7713f0d2e793d6516c71058: Status 404 returned error can't find the container with id 9bdfe3c8e44691b106ce3dc55115577c81f08f1ea7713f0d2e793d6516c71058 Sep 16 13:42:48.822819 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:48.822621 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:42:49.071058 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.070930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:49.071233 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:49.071086 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:42:49.071233 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:49.071182 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs podName:42514386-d0ca-4a5b-b5aa-d0e5b3276cd0 nodeName:}" failed. No retries permitted until 2026-09-16 13:42:50.071148376 +0000 UTC m=+4.034507203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs") pod "network-metrics-daemon-9dzbs" (UID: "42514386-d0ca-4a5b-b5aa-d0e5b3276cd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:42:49.172044 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.172005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:49.172225 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:49.172128 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:42:49.172312 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:49.172223 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret podName:1499dfa1-b0ce-44a0-add8-c1404440ecdd nodeName:}" failed. No retries permitted until 2026-09-16 13:42:50.172202369 +0000 UTC m=+4.135561180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret") pod "global-pull-secret-syncer-hddml" (UID: "1499dfa1-b0ce-44a0-add8-c1404440ecdd") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:42:49.272922 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.272886 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696t4\" (UniqueName: \"kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4\") pod \"network-check-target-mcl5r\" (UID: \"f0d7ecc6-7b55-4cca-9c55-093d59c1e854\") " pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:49.273100 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:49.273047 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:42:49.273100 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:49.273070 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:42:49.273100 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:49.273084 2581 projected.go:196] Error preparing data for projected volume kube-api-access-696t4 for pod openshift-network-diagnostics/network-check-target-mcl5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:42:49.273334 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:49.273149 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4 podName:f0d7ecc6-7b55-4cca-9c55-093d59c1e854 nodeName:}" failed. No retries permitted until 2026-09-16 13:42:50.273127817 +0000 UTC m=+4.236486627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-696t4" (UniqueName: "kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4") pod "network-check-target-mcl5r" (UID: "f0d7ecc6-7b55-4cca-9c55-093d59c1e854") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:42:49.501253 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.501138 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:37:48 +0000 UTC" deadline="2028-05-15 20:40:25.697170466 +0000 UTC" Sep 16 13:42:49.501253 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.501203 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14574h57m36.195971685s" Sep 16 13:42:49.519467 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.519437 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kzjdt" event={"ID":"ebd160ee-64b4-4ca0-9368-510cb8bab449","Type":"ContainerStarted","Data":"2d1b4a2890b035341f95c461315192615ab6fbbdcf5609c4c13c97b11543c9e0"} Sep 16 13:42:49.520883 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.520851 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n97z8" event={"ID":"f4305a9e-2784-4ecd-9462-64b45a220538","Type":"ContainerStarted","Data":"3421ed94d07a6c7d550380924bbc6eefab0792084d028e05a3bcdf52c944bcbf"} Sep 16 13:42:49.522228 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.522180 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" event={"ID":"740f47ff-eb88-4272-8677-19e1d3c2148f","Type":"ContainerStarted","Data":"d6388d9798ce6896592be9d290256e5f7063f5f480d75de73f835dd8e0d97882"} Sep 16 13:42:49.523553 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.523527 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" event={"ID":"23cd4ca2-227a-4011-ba43-5afafff1d0b9","Type":"ContainerStarted","Data":"a49d9382a76242dc0ce3800090ba42aa9e5abda869be0fb103661df4fe5f63b7"} Sep 16 13:42:49.524915 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.524881 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gvnzs" event={"ID":"cd6779bd-b278-4b06-a7ca-80a4aae73ae0","Type":"ContainerStarted","Data":"9bdfe3c8e44691b106ce3dc55115577c81f08f1ea7713f0d2e793d6516c71058"} Sep 16 13:42:49.526117 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.526096 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmd5t" event={"ID":"1543153d-e6e7-44f0-881e-9f78070b70c7","Type":"ContainerStarted","Data":"b207f4bc162ca7ac9f5dab91e925be08b0fb1274f4afffc2a373d4507db47de8"} Sep 16 13:42:49.527447 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.527411 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" event={"ID":"7c27e30f-2283-4a75-bffd-fe46cda06f05","Type":"ContainerStarted","Data":"6500d35389e18a980da9f4ebd0178ff279a279f58e9c2a538c8de79a1f99aa33"} Sep 16 13:42:49.528671 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.528649 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" event={"ID":"46165266-25c4-426d-9917-f0937f9df64a","Type":"ContainerStarted","Data":"3211e15fd5476de160a5c21cc8b6964e408454b8258a46c3989ceb15c5b31749"} Sep 16 13:42:49.529877 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:49.529849 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vt5fx" event={"ID":"97fc753e-b824-4150-b7e1-051cdd2992ed","Type":"ContainerStarted","Data":"c380b3e5e150f3a191a421e44ec14bcd495e7252cade560ba95743c0aca7d557"} Sep 16 13:42:50.079094 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:50.079044 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:50.079302 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.079212 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:42:50.079302 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.079276 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs podName:42514386-d0ca-4a5b-b5aa-d0e5b3276cd0 nodeName:}" failed. No retries permitted until 2026-09-16 13:42:52.079258553 +0000 UTC m=+6.042617377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs") pod "network-metrics-daemon-9dzbs" (UID: "42514386-d0ca-4a5b-b5aa-d0e5b3276cd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:42:50.118916 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:50.118886 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:42:50.179691 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:50.179656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:50.179857 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.179842 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:42:50.179915 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.179906 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret podName:1499dfa1-b0ce-44a0-add8-c1404440ecdd nodeName:}" failed. No retries permitted until 2026-09-16 13:42:52.179886373 +0000 UTC m=+6.143245185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret") pod "global-pull-secret-syncer-hddml" (UID: "1499dfa1-b0ce-44a0-add8-c1404440ecdd") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:42:50.280245 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:50.280193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696t4\" (UniqueName: \"kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4\") pod \"network-check-target-mcl5r\" (UID: \"f0d7ecc6-7b55-4cca-9c55-093d59c1e854\") " pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:50.280431 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.280405 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:42:50.280496 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.280441 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:42:50.280496 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.280457 2581 projected.go:196] Error preparing data for projected volume kube-api-access-696t4 for pod openshift-network-diagnostics/network-check-target-mcl5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:42:50.280599 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.280521 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4 podName:f0d7ecc6-7b55-4cca-9c55-093d59c1e854 nodeName:}" failed. No retries permitted until 2026-09-16 13:42:52.280502145 +0000 UTC m=+6.243860981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-696t4" (UniqueName: "kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4") pod "network-check-target-mcl5r" (UID: "f0d7ecc6-7b55-4cca-9c55-093d59c1e854") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:42:50.502187 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:50.502099 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:37:48 +0000 UTC" deadline="2028-03-10 16:55:34.456062047 +0000 UTC" Sep 16 13:42:50.502187 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:50.502141 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12987h12m43.953926537s" Sep 16 13:42:50.512264 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:50.509009 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:50.512264 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.509156 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:42:50.512264 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:50.509614 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:50.512264 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.509771 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:42:50.514361 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:50.512211 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:50.514361 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:50.513878 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:42:51.547818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:51.547772 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal" event={"ID":"039e38a7f72228dfa18bca5c2b7c263f","Type":"ContainerStarted","Data":"d65441420fb197f53fde4f7b1872d44f651d174a09c4355060af580b141d57b0"} Sep 16 13:42:51.553445 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:51.553413 2581 generic.go:356] "Generic (PLEG): container finished" podID="9ff7c77e7f1b706d9871c56b53d5c67e" containerID="f5a97ef96dd02a4c4e6e3568c2e75f8aa22f6834a5e6c26a56556f5399a635be" exitCode=0 Sep 16 13:42:51.553597 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:51.553464 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" event={"ID":"9ff7c77e7f1b706d9871c56b53d5c67e","Type":"ContainerDied","Data":"f5a97ef96dd02a4c4e6e3568c2e75f8aa22f6834a5e6c26a56556f5399a635be"} Sep 16 13:42:51.591195 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:51.590085 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-222.ec2.internal" podStartSLOduration=3.590062251 podStartE2EDuration="3.590062251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:42:51.569187795 +0000 UTC m=+5.532546627" watchObservedRunningTime="2026-09-16 13:42:51.590062251 +0000 UTC m=+5.553421083" Sep 16 13:42:52.098848 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:52.098807 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:52.099053 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.099010 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:42:52.099123 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.099083 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs podName:42514386-d0ca-4a5b-b5aa-d0e5b3276cd0 nodeName:}" failed. No retries permitted until 2026-09-16 13:42:56.099062207 +0000 UTC m=+10.062421020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs") pod "network-metrics-daemon-9dzbs" (UID: "42514386-d0ca-4a5b-b5aa-d0e5b3276cd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:42:52.200082 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:52.200014 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:52.200280 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.200170 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:42:52.200894 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.200854 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret podName:1499dfa1-b0ce-44a0-add8-c1404440ecdd nodeName:}" failed. No retries permitted until 2026-09-16 13:42:56.200818086 +0000 UTC m=+10.164176909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret") pod "global-pull-secret-syncer-hddml" (UID: "1499dfa1-b0ce-44a0-add8-c1404440ecdd") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:42:52.300506 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:52.300459 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696t4\" (UniqueName: \"kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4\") pod \"network-check-target-mcl5r\" (UID: \"f0d7ecc6-7b55-4cca-9c55-093d59c1e854\") " pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:52.300695 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.300680 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:42:52.300763 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.300702 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:42:52.300763 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.300715 2581 projected.go:196] Error preparing data for projected volume kube-api-access-696t4 for pod openshift-network-diagnostics/network-check-target-mcl5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:42:52.300875 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.300780 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4 podName:f0d7ecc6-7b55-4cca-9c55-093d59c1e854 nodeName:}" failed. No retries permitted until 2026-09-16 13:42:56.300760199 +0000 UTC m=+10.264119010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-696t4" (UniqueName: "kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4") pod "network-check-target-mcl5r" (UID: "f0d7ecc6-7b55-4cca-9c55-093d59c1e854") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:42:52.513228 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:52.512335 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:52.513228 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.512479 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:42:52.513228 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:52.512921 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:52.513228 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.513027 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:42:52.513596 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:52.513551 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:52.513701 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:52.513648 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:42:54.509284 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:54.509252 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:54.509817 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:54.509295 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:54.509817 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:54.509262 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:54.509817 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:54.509402 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:42:54.509817 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:54.509453 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:42:54.509817 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:54.509517 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:42:56.133665 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:56.133542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:56.134161 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.133675 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:42:56.134161 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.133740 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs podName:42514386-d0ca-4a5b-b5aa-d0e5b3276cd0 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:04.133721162 +0000 UTC m=+18.097079967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs") pod "network-metrics-daemon-9dzbs" (UID: "42514386-d0ca-4a5b-b5aa-d0e5b3276cd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:42:56.233952 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:56.233913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:56.234125 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.234050 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:42:56.234125 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.234111 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret podName:1499dfa1-b0ce-44a0-add8-c1404440ecdd nodeName:}" failed. No retries permitted until 2026-09-16 13:43:04.234093843 +0000 UTC m=+18.197452653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret") pod "global-pull-secret-syncer-hddml" (UID: "1499dfa1-b0ce-44a0-add8-c1404440ecdd") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:42:56.334581 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:56.334427 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696t4\" (UniqueName: \"kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4\") pod \"network-check-target-mcl5r\" (UID: \"f0d7ecc6-7b55-4cca-9c55-093d59c1e854\") " pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:56.334774 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.334661 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:42:56.334774 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.334682 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:42:56.334774 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.334695 2581 projected.go:196] Error preparing data for projected volume kube-api-access-696t4 for pod openshift-network-diagnostics/network-check-target-mcl5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:42:56.334774 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.334756 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4 podName:f0d7ecc6-7b55-4cca-9c55-093d59c1e854 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:04.334739564 +0000 UTC m=+18.298098389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-696t4" (UniqueName: "kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4") pod "network-check-target-mcl5r" (UID: "f0d7ecc6-7b55-4cca-9c55-093d59c1e854") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:42:56.510322 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:56.510066 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:56.510322 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.510211 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:42:56.510530 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:56.510378 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:56.510530 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:56.510433 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:56.510530 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.510508 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:42:56.510743 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:56.510572 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:42:58.510016 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:58.509985 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:42:58.510500 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:58.510033 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:42:58.510500 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:58.510123 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:42:58.510500 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:42:58.510382 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:42:58.510626 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:58.510496 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:42:58.510626 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:42:58.510584 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:00.509622 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:00.509424 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:00.510065 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:00.509424 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:00.510065 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:00.509722 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:00.510065 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:00.509784 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:43:00.510065 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:00.509500 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:00.510065 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:00.509880 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:43:02.509505 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:02.509467 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:02.509505 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:02.509498 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:02.510008 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:02.509468 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:02.510008 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:02.509598 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:43:02.510008 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:02.509686 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:02.510008 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:02.509787 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:43:04.190598 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:04.190567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:04.191105 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.190751 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:04.191105 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.190829 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs podName:42514386-d0ca-4a5b-b5aa-d0e5b3276cd0 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:20.190810346 +0000 UTC m=+34.154169151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs") pod "network-metrics-daemon-9dzbs" (UID: "42514386-d0ca-4a5b-b5aa-d0e5b3276cd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:04.291586 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:04.291560 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:04.291748 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.291670 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:04.291748 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.291725 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret podName:1499dfa1-b0ce-44a0-add8-c1404440ecdd nodeName:}" failed. No retries permitted until 2026-09-16 13:43:20.29171118 +0000 UTC m=+34.255069985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret") pod "global-pull-secret-syncer-hddml" (UID: "1499dfa1-b0ce-44a0-add8-c1404440ecdd") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:04.392659 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:04.392625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696t4\" (UniqueName: \"kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4\") pod \"network-check-target-mcl5r\" (UID: \"f0d7ecc6-7b55-4cca-9c55-093d59c1e854\") " pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:04.392830 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.392811 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:43:04.392909 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.392839 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:43:04.392909 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.392855 2581 projected.go:196] Error preparing data for projected volume kube-api-access-696t4 for pod openshift-network-diagnostics/network-check-target-mcl5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:04.392999 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.392911 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4 podName:f0d7ecc6-7b55-4cca-9c55-093d59c1e854 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:20.392896192 +0000 UTC m=+34.356255016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-696t4" (UniqueName: "kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4") pod "network-check-target-mcl5r" (UID: "f0d7ecc6-7b55-4cca-9c55-093d59c1e854") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:04.509232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:04.509196 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:04.509232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:04.509225 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:04.509453 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:04.509199 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:04.509453 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.509326 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:43:04.509539 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.509447 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:43:04.509603 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:04.509540 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:06.510760 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:06.510722 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:06.511255 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:06.510770 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:06.511255 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:06.510829 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:06.511255 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:06.510875 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:43:06.511255 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:06.510901 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:43:06.511255 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:06.510998 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:07.589918 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.589724 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" event={"ID":"7c27e30f-2283-4a75-bffd-fe46cda06f05","Type":"ContainerStarted","Data":"297905a0c0d51885e70abeaa6b9e51e232c049d7373da2f0b7f39320a0152a33"} Sep 16 13:43:07.591695 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.591672 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" event={"ID":"46165266-25c4-426d-9917-f0937f9df64a","Type":"ContainerStarted","Data":"bb6f9dcf633622e1a41c0a142b1eb0a632bdeee10464b853040ddc4a6320f9ef"} Sep 16 13:43:07.594156 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.593428 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n97z8" event={"ID":"f4305a9e-2784-4ecd-9462-64b45a220538","Type":"ContainerStarted","Data":"472db09ea46c8a1ef47f0bda9da75deeb8914640cf3b584a367a257ebb7c8ed3"} Sep 16 13:43:07.595222 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.595200 2581 generic.go:356] "Generic (PLEG): container finished" podID="740f47ff-eb88-4272-8677-19e1d3c2148f" containerID="f27526d204e1b94592cb55415e7d3d66f7bdc5730712b5fb615b808275f6c5f6" exitCode=0 Sep 16 13:43:07.595352 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.595278 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" event={"ID":"740f47ff-eb88-4272-8677-19e1d3c2148f","Type":"ContainerDied","Data":"f27526d204e1b94592cb55415e7d3d66f7bdc5730712b5fb615b808275f6c5f6"} Sep 16 13:43:07.598107 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.598011 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:43:07.598555 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.598460 2581 generic.go:356] "Generic (PLEG): container finished" podID="23cd4ca2-227a-4011-ba43-5afafff1d0b9" containerID="a626103b46520f39f0309672643a5bbf6f5e320a78920454d4f185117ca41a74" exitCode=1 Sep 16 13:43:07.598555 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.598485 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" event={"ID":"23cd4ca2-227a-4011-ba43-5afafff1d0b9","Type":"ContainerDied","Data":"a626103b46520f39f0309672643a5bbf6f5e320a78920454d4f185117ca41a74"} Sep 16 13:43:07.598695 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.598560 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" event={"ID":"23cd4ca2-227a-4011-ba43-5afafff1d0b9","Type":"ContainerStarted","Data":"16a9ef30479a0343fbe742687ee4cd28293f70376474fb663e015951c4b8d71e"} Sep 16 13:43:07.600926 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.600902 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" event={"ID":"9ff7c77e7f1b706d9871c56b53d5c67e","Type":"ContainerStarted","Data":"1425005b0e1b62d7635cb2b06ec341394c764257eeb0f155457b1d17b6578ec0"} Sep 16 13:43:07.602538 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.602519 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gvnzs" event={"ID":"cd6779bd-b278-4b06-a7ca-80a4aae73ae0","Type":"ContainerStarted","Data":"2b19427ad234a4028073be65e4286b4ebe5c4720164262e1f015032847928913"} Sep 16 13:43:07.603950 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.603933 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmd5t" event={"ID":"1543153d-e6e7-44f0-881e-9f78070b70c7","Type":"ContainerStarted","Data":"c4da4e220cd2c98d2b27222a947d1e58499572d76c805218a569d288bceb6a41"} Sep 16 13:43:07.609287 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.609241 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hnw8s" podStartSLOduration=2.212754973 podStartE2EDuration="20.609223877s" totalImagesPullingTime="18.396468904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:07.608929518 +0000 UTC m=+21.572288345" watchObservedRunningTime="2026-09-16 13:43:07.609223877 +0000 UTC m=+21.572582705" Sep 16 13:43:07.628506 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.628470 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-222.ec2.internal" podStartSLOduration=19.628454183 podStartE2EDuration="19.628454183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:07.627971602 +0000 UTC m=+21.591330424" watchObservedRunningTime="2026-09-16 13:43:07.628454183 +0000 UTC m=+21.591813010" Sep 16 13:43:07.678220 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.678029 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zmd5t" podStartSLOduration=2.277164521 podStartE2EDuration="20.67800767s" totalImagesPullingTime="18.400843149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:07.659600733 +0000 UTC m=+21.622959561" watchObservedRunningTime="2026-09-16 13:43:07.67800767 +0000 UTC m=+21.641366498" Sep 16 13:43:07.678811 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.678779 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gvnzs" podStartSLOduration=6.390808697 podStartE2EDuration="20.678767926s" totalImagesPullingTime="14.287959229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:07.678433431 +0000 UTC m=+21.641792287" watchObservedRunningTime="2026-09-16 13:43:07.678767926 +0000 UTC m=+21.642126752" Sep 16 13:43:07.724571 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:07.724528 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n97z8" podStartSLOduration=2.535692557 podStartE2EDuration="20.724513945s" totalImagesPullingTime="18.188821388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:07.724224554 +0000 UTC m=+21.687583384" watchObservedRunningTime="2026-09-16 13:43:07.724513945 +0000 UTC m=+21.687872772" Sep 16 13:43:08.509529 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.509503 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:08.509676 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.509503 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:08.509676 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:08.509622 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:08.509676 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.509644 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:08.509834 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:08.509737 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:43:08.509834 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:08.509805 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:43:08.596252 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.596221 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 13:43:08.608354 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.608321 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" event={"ID":"46165266-25c4-426d-9917-f0937f9df64a","Type":"ContainerStarted","Data":"f2291b38037f5013e57bb8bd7783ba0dc3f39a34be04509f6c843667154474d4"} Sep 16 13:43:08.609911 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.609885 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vt5fx" event={"ID":"97fc753e-b824-4150-b7e1-051cdd2992ed","Type":"ContainerStarted","Data":"568cbf0e48fcd4fe8886b7bcb05c71f4bcf4a33c0ccb66ab635ef6eaf6b67630"} Sep 16 13:43:08.611414 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.611388 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kzjdt" event={"ID":"ebd160ee-64b4-4ca0-9368-510cb8bab449","Type":"ContainerStarted","Data":"793100fc1e619153c0b27862211220ee76907f391cd93c7baa47e38bf91fb247"} Sep 16 13:43:08.614896 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.614881 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:43:08.615424 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.615400 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" event={"ID":"23cd4ca2-227a-4011-ba43-5afafff1d0b9","Type":"ContainerStarted","Data":"b91265650d66b8c7b0d2bde84444fdb142ca9517a551a01f3cfa80b06d84cc9f"} Sep 16 13:43:08.615536 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.615502 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" event={"ID":"23cd4ca2-227a-4011-ba43-5afafff1d0b9","Type":"ContainerStarted","Data":"9b4358821b57b56ddf976e9c092d52ee83280da836cf383e835827d2ad7ad348"} Sep 16 13:43:08.615621 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.615554 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" event={"ID":"23cd4ca2-227a-4011-ba43-5afafff1d0b9","Type":"ContainerStarted","Data":"7bf7bd696742d24bd4fe358fc7452bc60a1ec04860f4e847ce2afb234fe23268"} Sep 16 13:43:08.615621 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.615604 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" event={"ID":"23cd4ca2-227a-4011-ba43-5afafff1d0b9","Type":"ContainerStarted","Data":"8ad2e68321b3b1ddf037d19976ee1d69372628f1792c403773ad573e8fa86bf8"} Sep 16 13:43:08.630341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.630299 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vt5fx" podStartSLOduration=7.309769285 podStartE2EDuration="21.630280795s" totalImagesPullingTime="14.32051151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:08.629766983 +0000 UTC m=+22.593125812" watchObservedRunningTime="2026-09-16 13:43:08.630280795 +0000 UTC m=+22.593639627" Sep 16 13:43:08.655870 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:08.655820 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kzjdt" podStartSLOduration=2.817367023 podStartE2EDuration="21.655803725s" totalImagesPullingTime="18.838436702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:08.654864051 +0000 UTC m=+22.618222871" watchObservedRunningTime="2026-09-16 13:43:08.655803725 +0000 UTC m=+22.619162551" Sep 16 13:43:09.523181 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:09.523025 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T13:43:08.596243574Z","UUID":"6a492b78-2acf-4822-8d1a-77f004f1451c","Handler":null,"Name":"","Endpoint":""} Sep 16 13:43:09.524806 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:09.524780 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 13:43:09.524938 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:09.524818 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 13:43:10.508981 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:10.508950 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:10.509533 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:10.508950 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:10.509533 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:10.509067 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:43:10.509533 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:10.508954 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:10.509533 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:10.509171 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:10.509533 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:10.509268 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:43:10.622408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:10.622375 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" event={"ID":"46165266-25c4-426d-9917-f0937f9df64a","Type":"ContainerStarted","Data":"118c5348d86cbaa59f7f3fe2ab7f2f31c2e9f9481783bcf7aae6a039b7e9c978"} Sep 16 13:43:10.625281 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:10.625261 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:43:10.625604 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:10.625587 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" event={"ID":"23cd4ca2-227a-4011-ba43-5afafff1d0b9","Type":"ContainerStarted","Data":"e4c04df745e6f5a72938ff425b2171de6ecea3acfcb2a638810ad477f58b37cb"} Sep 16 13:43:10.643986 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:10.643943 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9j9l5" podStartSLOduration=3.273853461 podStartE2EDuration="23.643927795s" totalImagesPullingTime="20.370074334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:10.642476521 +0000 UTC m=+24.605835347" watchObservedRunningTime="2026-09-16 13:43:10.643927795 +0000 UTC m=+24.607286625" Sep 16 13:43:11.085553 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:11.085510 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:43:11.086149 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:11.086123 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:43:11.628472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:11.628443 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:43:11.628975 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:11.628952 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zmd5t" Sep 16 13:43:12.510167 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:12.509989 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:12.510272 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:12.510005 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:12.510339 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:12.510263 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:43:12.510339 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:12.510010 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:12.510339 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:12.510324 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:12.510499 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:12.510392 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:43:12.632819 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:12.632799 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:43:12.633325 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:12.633123 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" event={"ID":"23cd4ca2-227a-4011-ba43-5afafff1d0b9","Type":"ContainerStarted","Data":"a35d2618763543b2922294aa0b57e5cd95a989edde7847049dee872844522352"} Sep 16 13:43:12.633710 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:12.633694 2581 scope.go:117] "RemoveContainer" containerID="a626103b46520f39f0309672643a5bbf6f5e320a78920454d4f185117ca41a74" Sep 16 13:43:13.636514 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:13.636483 2581 generic.go:356] "Generic (PLEG): container finished" podID="740f47ff-eb88-4272-8677-19e1d3c2148f" containerID="cfefb048087fef1583dedb8bc27e7a35c7b18637bf0c857f8e24b21a4fb5f9ad" exitCode=0 Sep 16 13:43:13.637258 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:13.636555 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" event={"ID":"740f47ff-eb88-4272-8677-19e1d3c2148f","Type":"ContainerDied","Data":"cfefb048087fef1583dedb8bc27e7a35c7b18637bf0c857f8e24b21a4fb5f9ad"} Sep 16 13:43:13.640628 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:13.640609 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:43:13.641022 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:13.641004 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" event={"ID":"23cd4ca2-227a-4011-ba43-5afafff1d0b9","Type":"ContainerStarted","Data":"77b00632ab73cfcfe52b26416571a620dc546b7d329eb52a071f5897888d03f0"} Sep 16 13:43:13.641304 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:13.641289 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:43:13.641399 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:13.641339 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:43:13.641399 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:13.641380 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:43:13.656155 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:13.656117 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:43:13.656350 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:13.656335 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:43:13.695101 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:13.694893 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" podStartSLOduration=8.21874353 podStartE2EDuration="26.694875162s" totalImagesPullingTime="18.476131632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:13.694715465 +0000 UTC m=+27.658074294" watchObservedRunningTime="2026-09-16 13:43:13.694875162 +0000 UTC m=+27.658233990" Sep 16 13:43:14.467931 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:14.467716 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hddml"] Sep 16 13:43:14.468118 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:14.468062 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:14.468199 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:14.468181 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:14.470845 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:14.470817 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mcl5r"] Sep 16 13:43:14.470976 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:14.470916 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:14.471043 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:14.471021 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:43:14.473826 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:14.473793 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9dzbs"] Sep 16 13:43:14.473952 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:14.473919 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:14.474039 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:14.474014 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:43:15.647813 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:15.647725 2581 generic.go:356] "Generic (PLEG): container finished" podID="740f47ff-eb88-4272-8677-19e1d3c2148f" containerID="cd227d64dd6089ffda8454adee922eb985aca2a0d1e45d40abec0a24d225c192" exitCode=0 Sep 16 13:43:15.647813 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:15.647792 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" event={"ID":"740f47ff-eb88-4272-8677-19e1d3c2148f","Type":"ContainerDied","Data":"cd227d64dd6089ffda8454adee922eb985aca2a0d1e45d40abec0a24d225c192"} Sep 16 13:43:16.510715 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:16.510685 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:16.510860 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:16.510782 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:16.510911 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:16.510874 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:16.511003 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:16.510983 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:43:16.511057 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:16.511047 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:16.511121 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:16.511108 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:43:17.653882 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:17.653850 2581 generic.go:356] "Generic (PLEG): container finished" podID="740f47ff-eb88-4272-8677-19e1d3c2148f" containerID="d753e7e67367c2bef0e9044fa98ed9971b335a68ede06b686d481d1454672203" exitCode=0 Sep 16 13:43:17.654251 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:17.653903 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" event={"ID":"740f47ff-eb88-4272-8677-19e1d3c2148f","Type":"ContainerDied","Data":"d753e7e67367c2bef0e9044fa98ed9971b335a68ede06b686d481d1454672203"} Sep 16 13:43:18.509812 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:18.509783 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:18.509983 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:18.509899 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hddml" podUID="1499dfa1-b0ce-44a0-add8-c1404440ecdd" Sep 16 13:43:18.509983 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:18.509783 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:18.509983 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:18.509927 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:18.510158 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:18.510002 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dzbs" podUID="42514386-d0ca-4a5b-b5aa-d0e5b3276cd0" Sep 16 13:43:18.510158 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:18.510072 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcl5r" podUID="f0d7ecc6-7b55-4cca-9c55-093d59c1e854" Sep 16 13:43:20.207239 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.207060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:20.207638 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.207220 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:20.207638 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.207373 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs podName:42514386-d0ca-4a5b-b5aa-d0e5b3276cd0 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:52.207349718 +0000 UTC m=+66.170708523 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs") pod "network-metrics-daemon-9dzbs" (UID: "42514386-d0ca-4a5b-b5aa-d0e5b3276cd0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:43:20.308578 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.308502 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:20.308728 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.308629 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:20.308728 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.308690 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret podName:1499dfa1-b0ce-44a0-add8-c1404440ecdd nodeName:}" failed. No retries permitted until 2026-09-16 13:43:52.308671784 +0000 UTC m=+66.272030607 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret") pod "global-pull-secret-syncer-hddml" (UID: "1499dfa1-b0ce-44a0-add8-c1404440ecdd") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:43:20.384970 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.384947 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-222.ec2.internal" event="NodeReady" Sep 16 13:43:20.385131 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.385052 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 13:43:20.409610 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.409583 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696t4\" (UniqueName: \"kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4\") pod \"network-check-target-mcl5r\" (UID: \"f0d7ecc6-7b55-4cca-9c55-093d59c1e854\") " pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:20.409743 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.409700 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:43:20.409743 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.409718 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:43:20.409743 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.409733 2581 projected.go:196] Error preparing data for projected volume kube-api-access-696t4 for pod openshift-network-diagnostics/network-check-target-mcl5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:20.409866 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.409803 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4 podName:f0d7ecc6-7b55-4cca-9c55-093d59c1e854 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:52.409771902 +0000 UTC m=+66.373130711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-696t4" (UniqueName: "kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4") pod "network-check-target-mcl5r" (UID: "f0d7ecc6-7b55-4cca-9c55-093d59c1e854") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:43:20.424471 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.424447 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4"] Sep 16 13:43:20.499206 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.499176 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67fcbbf884-59lgn"] Sep 16 13:43:20.499351 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.499278 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:20.501860 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.501791 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 16 13:43:20.501860 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.501811 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:43:20.501860 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.501802 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-97tnf\"" Sep 16 13:43:20.502101 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.501834 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 16 13:43:20.513679 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.513618 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr"] Sep 16 13:43:20.513785 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.513694 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:20.513785 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.513699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:20.513908 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.513893 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.514340 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.514160 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:20.516302 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.516279 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 13:43:20.516405 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.516305 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 13:43:20.516405 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.516380 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 13:43:20.516405 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.516391 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 13:43:20.516555 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.516509 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 13:43:20.516555 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.516528 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4knh2\"" Sep 16 13:43:20.516555 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.516531 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 13:43:20.516704 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.516621 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7xrvw\"" Sep 16 13:43:20.516704 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.516628 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s4xvt\"" Sep 16 13:43:20.516847 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.516831 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 13:43:20.521945 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.521845 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 13:43:20.527727 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.527707 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:20.530833 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.530812 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 13:43:20.531226 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.531210 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 16 13:43:20.534165 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.531330 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zs72k\"" Sep 16 13:43:20.534165 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.531832 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 16 13:43:20.534165 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.532057 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 13:43:20.536096 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.536077 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-6nvrv"] Sep 16 13:43:20.554985 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.554953 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct"] Sep 16 13:43:20.555094 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.555076 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.557192 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.557174 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 16 13:43:20.557554 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.557537 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ltpjg\"" Sep 16 13:43:20.557664 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.557583 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 13:43:20.557834 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.557819 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 13:43:20.557908 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.557828 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 16 13:43:20.563823 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.563807 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 16 13:43:20.573754 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.573739 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct" Sep 16 13:43:20.573892 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.573738 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-bcdcbb9b-kbf6r"] Sep 16 13:43:20.576419 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.576399 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 16 13:43:20.576509 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.576419 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:43:20.576509 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.576486 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tz7cd\"" Sep 16 13:43:20.593200 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.593183 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-qsvk6"] Sep 16 13:43:20.593309 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.593295 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.595909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.595890 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 16 13:43:20.596016 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.595915 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 16 13:43:20.596016 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.596009 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 13:43:20.596121 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.596013 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6szcg\"" Sep 16 13:43:20.596200 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.596133 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 16 13:43:20.596254 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.596199 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 16 13:43:20.596254 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.596201 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 13:43:20.606053 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.606029 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4"] Sep 16 13:43:20.606163 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.606068 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-qsvk6"] Sep 16 13:43:20.606163 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.606087 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-6nvrv"] Sep 16 13:43:20.606163 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.606107 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67fcbbf884-59lgn"] Sep 16 13:43:20.606163 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.606127 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-bcdcbb9b-kbf6r"] Sep 16 13:43:20.606355 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.606185 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.606355 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.606221 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf"] Sep 16 13:43:20.608387 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.608320 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 16 13:43:20.608484 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.608406 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 16 13:43:20.608484 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.608424 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 16 13:43:20.608484 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.608424 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lgbtn\"" Sep 16 13:43:20.608661 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.608555 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:43:20.610817 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.610789 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-image-registry-private-configuration\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.610914 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.610820 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-bound-sa-token\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.610914 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.610842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cph2f\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-kube-api-access-cph2f\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.610914 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.610874 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2f131fb-a923-428d-82eb-39376dbd5594-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:20.610914 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.610908 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4l45\" (UniqueName: \"kubernetes.io/projected/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-kube-api-access-m4l45\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:20.611123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.610956 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-trusted-ca\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.611123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.610977 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:20.611123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.611018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-certificates\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.611123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.611036 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:20.611123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.611065 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.611123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.611087 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28371faa-0a08-49cf-8a63-9abe68aca3bc-ca-trust-extracted\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.611491 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.611172 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-installation-pull-secrets\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.611491 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.611204 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmpd\" (UniqueName: \"kubernetes.io/projected/e2f131fb-a923-428d-82eb-39376dbd5594-kube-api-access-jxmpd\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:20.616914 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.616848 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 16 13:43:20.626440 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.626417 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t2dsl"] Sep 16 13:43:20.626542 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.626487 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.628754 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.628733 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qn8nm\"" Sep 16 13:43:20.628922 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.628905 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 16 13:43:20.628993 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.628959 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:43:20.628993 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.628988 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 16 13:43:20.629242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.629224 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 16 13:43:20.646125 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.646105 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg"] Sep 16 13:43:20.646254 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.646232 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.648889 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.648856 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 13:43:20.648992 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.648909 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 13:43:20.648992 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.648945 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kqx7z\"" Sep 16 13:43:20.666228 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.666209 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-52kgm"] Sep 16 13:43:20.666326 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.666316 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.668848 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.668828 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 16 13:43:20.668954 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.668833 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 16 13:43:20.669041 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.669024 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 16 13:43:20.669125 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.669116 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7czwx\"" Sep 16 13:43:20.669224 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.669202 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:43:20.684824 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.684807 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5brjf"] Sep 16 13:43:20.684945 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.684929 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-52kgm" Sep 16 13:43:20.687548 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.687531 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vk689\"" Sep 16 13:43:20.695736 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.695719 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg"] Sep 16 13:43:20.695825 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.695746 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct"] Sep 16 13:43:20.695825 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.695762 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-52kgm"] Sep 16 13:43:20.695825 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.695779 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr"] Sep 16 13:43:20.695825 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.695797 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf"] Sep 16 13:43:20.695825 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.695815 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2dsl"] Sep 16 13:43:20.696045 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.695830 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5brjf" Sep 16 13:43:20.696045 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.695839 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5brjf"] Sep 16 13:43:20.698079 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.698053 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 13:43:20.698198 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.698083 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n4dzm\"" Sep 16 13:43:20.698198 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.698099 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 13:43:20.698198 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.698084 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 13:43:20.711887 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.711871 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97f4c753-e774-452c-8c8d-4070badd940e-snapshots\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.712017 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.711895 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:20.712017 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.711911 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.712017 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.711930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glps5\" (UniqueName: \"kubernetes.io/projected/8736f18a-8bcd-4118-a84b-8d6526243746-kube-api-access-glps5\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.712017 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.711950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-certificates\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.712017 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.711973 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-default-certificate\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.712283 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.712015 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:43:20.712283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-stats-auth\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.712283 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.712072 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls podName:de3aec25-00f6-4571-bedb-0e4f2a1dfdba nodeName:}" failed. No retries permitted until 2026-09-16 13:43:21.212054089 +0000 UTC m=+35.175412898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rnhz4" (UID: "de3aec25-00f6-4571-bedb-0e4f2a1dfdba") : secret "samples-operator-tls" not found Sep 16 13:43:20.712283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-installation-pull-secrets\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.712283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2vld\" (UniqueName: \"kubernetes.io/projected/97f4c753-e774-452c-8c8d-4070badd940e-kube-api-access-t2vld\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.712283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.712283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8736f18a-8bcd-4118-a84b-8d6526243746-serving-cert\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.712283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712227 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-bound-sa-token\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.712283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712251 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97f4c753-e774-452c-8c8d-4070badd940e-serving-cert\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.712283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cph2f\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-kube-api-access-cph2f\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.712302 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712304 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4l45\" (UniqueName: \"kubernetes.io/projected/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-kube-api-access-m4l45\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.712314 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fcbbf884-59lgn: secret "image-registry-tls" not found Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712335 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-trusted-ca\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.712363 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls podName:28371faa-0a08-49cf-8a63-9abe68aca3bc nodeName:}" failed. No retries permitted until 2026-09-16 13:43:21.212349136 +0000 UTC m=+35.175707955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls") pod "image-registry-67fcbbf884-59lgn" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc") : secret "image-registry-tls" not found Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8736f18a-8bcd-4118-a84b-8d6526243746-trusted-ca\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712415 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f4c753-e774-452c-8c8d-4070badd940e-service-ca-bundle\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f4c753-e774-452c-8c8d-4070badd940e-trusted-ca-bundle\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712476 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgzhv\" (UniqueName: \"kubernetes.io/projected/9590e744-474c-4da6-a84f-eee73056b9c9-kube-api-access-fgzhv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69jqf\" (UID: \"9590e744-474c-4da6-a84f-eee73056b9c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712530 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-certificates\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwwm\" (UniqueName: \"kubernetes.io/projected/9c5c390b-795a-4aab-b5b0-b4dad8c51074-kube-api-access-wdwwm\") pod \"volume-data-source-validator-86695b7cc7-hslct\" (UID: \"9c5c390b-795a-4aab-b5b0-b4dad8c51074\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712614 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28371faa-0a08-49cf-8a63-9abe68aca3bc-ca-trust-extracted\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.712706 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.712668 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.712730 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls podName:e2f131fb-a923-428d-82eb-39376dbd5594 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:21.212713475 +0000 UTC m=+35.176072281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gtcr" (UID: "e2f131fb-a923-428d-82eb-39376dbd5594") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvv7m\" (UniqueName: \"kubernetes.io/projected/5520568f-3a4f-4665-b75c-d4726ccfebb1-kube-api-access-pvv7m\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmpd\" (UniqueName: \"kubernetes.io/projected/e2f131fb-a923-428d-82eb-39376dbd5594-kube-api-access-jxmpd\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.712902 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9590e744-474c-4da6-a84f-eee73056b9c9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69jqf\" (UID: \"9590e744-474c-4da6-a84f-eee73056b9c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.713000 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28371faa-0a08-49cf-8a63-9abe68aca3bc-ca-trust-extracted\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.713060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-image-registry-private-configuration\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.713100 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8736f18a-8bcd-4118-a84b-8d6526243746-config\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.713132 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2f131fb-a923-428d-82eb-39376dbd5594-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.713181 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9590e744-474c-4da6-a84f-eee73056b9c9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69jqf\" (UID: \"9590e744-474c-4da6-a84f-eee73056b9c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.713207 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97f4c753-e774-452c-8c8d-4070badd940e-tmp\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.713341 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.713258 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-trusted-ca\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.713916 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.713897 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2f131fb-a923-428d-82eb-39376dbd5594-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:20.716826 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.716808 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-installation-pull-secrets\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.716934 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.716915 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-image-registry-private-configuration\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.722381 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.722357 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-bound-sa-token\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.722816 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.722792 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmpd\" (UniqueName: \"kubernetes.io/projected/e2f131fb-a923-428d-82eb-39376dbd5594-kube-api-access-jxmpd\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:20.723518 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.723497 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4l45\" (UniqueName: \"kubernetes.io/projected/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-kube-api-access-m4l45\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:20.724061 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.724022 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cph2f\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-kube-api-access-cph2f\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:20.813990 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.813927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-default-certificate\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.813990 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.813960 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-stats-auth\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.813990 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.813980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2vld\" (UniqueName: \"kubernetes.io/projected/97f4c753-e774-452c-8c8d-4070badd940e-kube-api-access-t2vld\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.814242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.813999 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c00f5f-761c-4d82-a657-52e437aee43d-serving-cert\") pod \"service-ca-operator-cf7c776b7-bknfg\" (UID: \"80c00f5f-761c-4d82-a657-52e437aee43d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.814242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bglwg\" (UniqueName: \"kubernetes.io/projected/80c00f5f-761c-4d82-a657-52e437aee43d-kube-api-access-bglwg\") pod \"service-ca-operator-cf7c776b7-bknfg\" (UID: \"80c00f5f-761c-4d82-a657-52e437aee43d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.814242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814135 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8736f18a-8bcd-4118-a84b-8d6526243746-serving-cert\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.814242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814175 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzr89\" (UniqueName: \"kubernetes.io/projected/039c92e5-b92f-4090-867c-5ab2dd1ed94c-kube-api-access-rzr89\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.814242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97f4c753-e774-452c-8c8d-4070badd940e-serving-cert\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.814242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814219 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80c00f5f-761c-4d82-a657-52e437aee43d-config\") pod \"service-ca-operator-cf7c776b7-bknfg\" (UID: \"80c00f5f-761c-4d82-a657-52e437aee43d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.814547 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814266 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8736f18a-8bcd-4118-a84b-8d6526243746-trusted-ca\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.814547 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814299 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f4c753-e774-452c-8c8d-4070badd940e-service-ca-bundle\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.814547 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814320 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f4c753-e774-452c-8c8d-4070badd940e-trusted-ca-bundle\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.814547 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814345 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzhv\" (UniqueName: \"kubernetes.io/projected/9590e744-474c-4da6-a84f-eee73056b9c9-kube-api-access-fgzhv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69jqf\" (UID: \"9590e744-474c-4da6-a84f-eee73056b9c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.814547 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814407 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5zgx\" (UniqueName: \"kubernetes.io/projected/d8002868-fbfd-4166-9b6e-cb7e79f4d499-kube-api-access-x5zgx\") pod \"network-check-source-6f8b768455-52kgm\" (UID: \"d8002868-fbfd-4166-9b6e-cb7e79f4d499\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-52kgm" Sep 16 13:43:20.814547 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwwm\" (UniqueName: \"kubernetes.io/projected/9c5c390b-795a-4aab-b5b0-b4dad8c51074-kube-api-access-wdwwm\") pod \"volume-data-source-validator-86695b7cc7-hslct\" (UID: \"9c5c390b-795a-4aab-b5b0-b4dad8c51074\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct" Sep 16 13:43:20.814547 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814483 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvv7m\" (UniqueName: \"kubernetes.io/projected/5520568f-3a4f-4665-b75c-d4726ccfebb1-kube-api-access-pvv7m\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.814547 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.814923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9590e744-474c-4da6-a84f-eee73056b9c9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69jqf\" (UID: \"9590e744-474c-4da6-a84f-eee73056b9c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.814923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814595 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj7gn\" (UniqueName: \"kubernetes.io/projected/15c5b25d-bb84-4667-b18e-30771d4c1c25-kube-api-access-fj7gn\") pod \"ingress-canary-5brjf\" (UID: \"15c5b25d-bb84-4667-b18e-30771d4c1c25\") " pod="openshift-ingress-canary/ingress-canary-5brjf" Sep 16 13:43:20.814923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8736f18a-8bcd-4118-a84b-8d6526243746-config\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.814923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9590e744-474c-4da6-a84f-eee73056b9c9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69jqf\" (UID: \"9590e744-474c-4da6-a84f-eee73056b9c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.814923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814726 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15c5b25d-bb84-4667-b18e-30771d4c1c25-cert\") pod \"ingress-canary-5brjf\" (UID: \"15c5b25d-bb84-4667-b18e-30771d4c1c25\") " pod="openshift-ingress-canary/ingress-canary-5brjf" Sep 16 13:43:20.814923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97f4c753-e774-452c-8c8d-4070badd940e-tmp\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.814923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/039c92e5-b92f-4090-867c-5ab2dd1ed94c-config-volume\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.814923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814850 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97f4c753-e774-452c-8c8d-4070badd940e-snapshots\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.814923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.815395 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.815395 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814953 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f4c753-e774-452c-8c8d-4070badd940e-service-ca-bundle\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.815395 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.814993 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/039c92e5-b92f-4090-867c-5ab2dd1ed94c-tmp-dir\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.815395 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.815029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glps5\" (UniqueName: \"kubernetes.io/projected/8736f18a-8bcd-4118-a84b-8d6526243746-kube-api-access-glps5\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.816639 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.816128 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:43:20.816639 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.816210 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:21.316189734 +0000 UTC m=+35.279548555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : secret "router-metrics-certs-default" not found Sep 16 13:43:20.816639 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.816289 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8736f18a-8bcd-4118-a84b-8d6526243746-trusted-ca\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.816639 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.816441 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:21.316425875 +0000 UTC m=+35.279784700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : configmap references non-existent config key: service-ca.crt Sep 16 13:43:20.816942 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.816868 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-default-certificate\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.816942 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.816914 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9590e744-474c-4da6-a84f-eee73056b9c9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69jqf\" (UID: \"9590e744-474c-4da6-a84f-eee73056b9c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.817044 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.816958 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97f4c753-e774-452c-8c8d-4070badd940e-serving-cert\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.817318 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.817266 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8736f18a-8bcd-4118-a84b-8d6526243746-config\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.817428 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.817334 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97f4c753-e774-452c-8c8d-4070badd940e-tmp\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.817428 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.817360 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8736f18a-8bcd-4118-a84b-8d6526243746-serving-cert\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.817428 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.817411 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97f4c753-e774-452c-8c8d-4070badd940e-snapshots\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.817589 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.817498 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-stats-auth\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.817729 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.817705 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f4c753-e774-452c-8c8d-4070badd940e-trusted-ca-bundle\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.817942 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.817926 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9590e744-474c-4da6-a84f-eee73056b9c9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69jqf\" (UID: \"9590e744-474c-4da6-a84f-eee73056b9c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.827032 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.827008 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glps5\" (UniqueName: \"kubernetes.io/projected/8736f18a-8bcd-4118-a84b-8d6526243746-kube-api-access-glps5\") pod \"console-operator-696884fb9c-qsvk6\" (UID: \"8736f18a-8bcd-4118-a84b-8d6526243746\") " pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.827128 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.827092 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2vld\" (UniqueName: \"kubernetes.io/projected/97f4c753-e774-452c-8c8d-4070badd940e-kube-api-access-t2vld\") pod \"insights-operator-5844b856db-6nvrv\" (UID: \"97f4c753-e774-452c-8c8d-4070badd940e\") " pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.828115 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.828096 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzhv\" (UniqueName: \"kubernetes.io/projected/9590e744-474c-4da6-a84f-eee73056b9c9-kube-api-access-fgzhv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69jqf\" (UID: \"9590e744-474c-4da6-a84f-eee73056b9c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.828835 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.828814 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvv7m\" (UniqueName: \"kubernetes.io/projected/5520568f-3a4f-4665-b75c-d4726ccfebb1-kube-api-access-pvv7m\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:20.828939 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.828922 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwwm\" (UniqueName: \"kubernetes.io/projected/9c5c390b-795a-4aab-b5b0-b4dad8c51074-kube-api-access-wdwwm\") pod \"volume-data-source-validator-86695b7cc7-hslct\" (UID: \"9c5c390b-795a-4aab-b5b0-b4dad8c51074\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct" Sep 16 13:43:20.864311 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.864290 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-6nvrv" Sep 16 13:43:20.881706 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.881690 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct" Sep 16 13:43:20.915727 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.915697 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzr89\" (UniqueName: \"kubernetes.io/projected/039c92e5-b92f-4090-867c-5ab2dd1ed94c-kube-api-access-rzr89\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.915814 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.915742 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80c00f5f-761c-4d82-a657-52e437aee43d-config\") pod \"service-ca-operator-cf7c776b7-bknfg\" (UID: \"80c00f5f-761c-4d82-a657-52e437aee43d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.915814 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.915795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5zgx\" (UniqueName: \"kubernetes.io/projected/d8002868-fbfd-4166-9b6e-cb7e79f4d499-kube-api-access-x5zgx\") pod \"network-check-source-6f8b768455-52kgm\" (UID: \"d8002868-fbfd-4166-9b6e-cb7e79f4d499\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-52kgm" Sep 16 13:43:20.915901 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.915833 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj7gn\" (UniqueName: \"kubernetes.io/projected/15c5b25d-bb84-4667-b18e-30771d4c1c25-kube-api-access-fj7gn\") pod \"ingress-canary-5brjf\" (UID: \"15c5b25d-bb84-4667-b18e-30771d4c1c25\") " pod="openshift-ingress-canary/ingress-canary-5brjf" Sep 16 13:43:20.915901 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.915865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15c5b25d-bb84-4667-b18e-30771d4c1c25-cert\") pod \"ingress-canary-5brjf\" (UID: \"15c5b25d-bb84-4667-b18e-30771d4c1c25\") " pod="openshift-ingress-canary/ingress-canary-5brjf" Sep 16 13:43:20.915901 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.915895 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/039c92e5-b92f-4090-867c-5ab2dd1ed94c-config-volume\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.916041 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.915932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.916099 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.916034 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/039c92e5-b92f-4090-867c-5ab2dd1ed94c-tmp-dir\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.916099 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.916052 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:43:20.916099 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.916088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c00f5f-761c-4d82-a657-52e437aee43d-serving-cert\") pod \"service-ca-operator-cf7c776b7-bknfg\" (UID: \"80c00f5f-761c-4d82-a657-52e437aee43d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.916099 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:20.916100 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls podName:039c92e5-b92f-4090-867c-5ab2dd1ed94c nodeName:}" failed. No retries permitted until 2026-09-16 13:43:21.416083988 +0000 UTC m=+35.379442808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls") pod "dns-default-t2dsl" (UID: "039c92e5-b92f-4090-867c-5ab2dd1ed94c") : secret "dns-default-metrics-tls" not found Sep 16 13:43:20.916325 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.916126 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bglwg\" (UniqueName: \"kubernetes.io/projected/80c00f5f-761c-4d82-a657-52e437aee43d-kube-api-access-bglwg\") pod \"service-ca-operator-cf7c776b7-bknfg\" (UID: \"80c00f5f-761c-4d82-a657-52e437aee43d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.916378 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.916347 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/039c92e5-b92f-4090-867c-5ab2dd1ed94c-tmp-dir\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.916657 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.916628 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/039c92e5-b92f-4090-867c-5ab2dd1ed94c-config-volume\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.916753 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.916733 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80c00f5f-761c-4d82-a657-52e437aee43d-config\") pod \"service-ca-operator-cf7c776b7-bknfg\" (UID: \"80c00f5f-761c-4d82-a657-52e437aee43d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.916886 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.916871 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:20.918281 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.918256 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c00f5f-761c-4d82-a657-52e437aee43d-serving-cert\") pod \"service-ca-operator-cf7c776b7-bknfg\" (UID: \"80c00f5f-761c-4d82-a657-52e437aee43d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.918462 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.918445 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15c5b25d-bb84-4667-b18e-30771d4c1c25-cert\") pod \"ingress-canary-5brjf\" (UID: \"15c5b25d-bb84-4667-b18e-30771d4c1c25\") " pod="openshift-ingress-canary/ingress-canary-5brjf" Sep 16 13:43:20.929513 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.929486 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5zgx\" (UniqueName: \"kubernetes.io/projected/d8002868-fbfd-4166-9b6e-cb7e79f4d499-kube-api-access-x5zgx\") pod \"network-check-source-6f8b768455-52kgm\" (UID: \"d8002868-fbfd-4166-9b6e-cb7e79f4d499\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-52kgm" Sep 16 13:43:20.930027 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.930007 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj7gn\" (UniqueName: \"kubernetes.io/projected/15c5b25d-bb84-4667-b18e-30771d4c1c25-kube-api-access-fj7gn\") pod \"ingress-canary-5brjf\" (UID: \"15c5b25d-bb84-4667-b18e-30771d4c1c25\") " pod="openshift-ingress-canary/ingress-canary-5brjf" Sep 16 13:43:20.930239 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.930218 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bglwg\" (UniqueName: \"kubernetes.io/projected/80c00f5f-761c-4d82-a657-52e437aee43d-kube-api-access-bglwg\") pod \"service-ca-operator-cf7c776b7-bknfg\" (UID: \"80c00f5f-761c-4d82-a657-52e437aee43d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.930334 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.930253 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzr89\" (UniqueName: \"kubernetes.io/projected/039c92e5-b92f-4090-867c-5ab2dd1ed94c-kube-api-access-rzr89\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:20.936077 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.936060 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" Sep 16 13:43:20.976090 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.976069 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" Sep 16 13:43:20.994279 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:20.994254 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-52kgm" Sep 16 13:43:21.005411 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:21.005391 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5brjf" Sep 16 13:43:21.218900 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:21.218869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:21.219325 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:21.218946 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:21.219325 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:21.219018 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:21.219325 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.219043 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:43:21.219325 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.219069 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fcbbf884-59lgn: secret "image-registry-tls" not found Sep 16 13:43:21.219325 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.219105 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:43:21.219325 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.219137 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls podName:28371faa-0a08-49cf-8a63-9abe68aca3bc nodeName:}" failed. No retries permitted until 2026-09-16 13:43:22.219118328 +0000 UTC m=+36.182477140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls") pod "image-registry-67fcbbf884-59lgn" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc") : secret "image-registry-tls" not found Sep 16 13:43:21.219325 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.219135 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:21.219325 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.219174 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls podName:de3aec25-00f6-4571-bedb-0e4f2a1dfdba nodeName:}" failed. No retries permitted until 2026-09-16 13:43:22.219162444 +0000 UTC m=+36.182521258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rnhz4" (UID: "de3aec25-00f6-4571-bedb-0e4f2a1dfdba") : secret "samples-operator-tls" not found Sep 16 13:43:21.219325 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.219242 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls podName:e2f131fb-a923-428d-82eb-39376dbd5594 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:22.21922033 +0000 UTC m=+36.182579139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gtcr" (UID: "e2f131fb-a923-428d-82eb-39376dbd5594") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:21.320416 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:21.320378 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:21.320566 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:21.320520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:21.320566 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.320519 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:43:21.320687 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.320620 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:22.320601432 +0000 UTC m=+36.283960242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : secret "router-metrics-certs-default" not found Sep 16 13:43:21.320752 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.320718 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:22.320697918 +0000 UTC m=+36.284056724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : configmap references non-existent config key: service-ca.crt Sep 16 13:43:21.421173 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:21.421130 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:21.421351 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.421329 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:43:21.421472 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:21.421385 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls podName:039c92e5-b92f-4090-867c-5ab2dd1ed94c nodeName:}" failed. No retries permitted until 2026-09-16 13:43:22.421371748 +0000 UTC m=+36.384730553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls") pod "dns-default-t2dsl" (UID: "039c92e5-b92f-4090-867c-5ab2dd1ed94c") : secret "dns-default-metrics-tls" not found Sep 16 13:43:22.227878 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.227835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:22.228411 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.227955 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:22.228411 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.227994 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:22.228411 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.228005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:22.228411 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.228063 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls podName:e2f131fb-a923-428d-82eb-39376dbd5594 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:24.228047855 +0000 UTC m=+38.191406666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gtcr" (UID: "e2f131fb-a923-428d-82eb-39376dbd5594") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:22.228411 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.228095 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:43:22.228411 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.228122 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:43:22.228411 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.228135 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fcbbf884-59lgn: secret "image-registry-tls" not found Sep 16 13:43:22.228411 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.228190 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls podName:de3aec25-00f6-4571-bedb-0e4f2a1dfdba nodeName:}" failed. No retries permitted until 2026-09-16 13:43:24.228172416 +0000 UTC m=+38.191531235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rnhz4" (UID: "de3aec25-00f6-4571-bedb-0e4f2a1dfdba") : secret "samples-operator-tls" not found Sep 16 13:43:22.228411 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.228209 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls podName:28371faa-0a08-49cf-8a63-9abe68aca3bc nodeName:}" failed. No retries permitted until 2026-09-16 13:43:24.228200241 +0000 UTC m=+38.191559050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls") pod "image-registry-67fcbbf884-59lgn" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc") : secret "image-registry-tls" not found Sep 16 13:43:22.329081 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.329054 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:22.329280 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.329161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:22.329280 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.329246 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:24.329226468 +0000 UTC m=+38.292585293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : configmap references non-existent config key: service-ca.crt Sep 16 13:43:22.329387 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.329291 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:43:22.329387 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.329362 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:24.32934481 +0000 UTC m=+38.292703634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : secret "router-metrics-certs-default" not found Sep 16 13:43:22.430361 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.430330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:22.430534 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.430476 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:43:22.430534 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:22.430533 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls podName:039c92e5-b92f-4090-867c-5ab2dd1ed94c nodeName:}" failed. No retries permitted until 2026-09-16 13:43:24.43051976 +0000 UTC m=+38.393878565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls") pod "dns-default-t2dsl" (UID: "039c92e5-b92f-4090-867c-5ab2dd1ed94c") : secret "dns-default-metrics-tls" not found Sep 16 13:43:22.762849 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.762646 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-6nvrv"] Sep 16 13:43:22.765549 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.765525 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg"] Sep 16 13:43:22.766210 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:22.766106 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f4c753_e774_452c_8c8d_4070badd940e.slice/crio-342573c5a2a5736a18b7a9b2a4001d110e2f930c2e6163ccb2691cfe79869e8a WatchSource:0}: Error finding container 342573c5a2a5736a18b7a9b2a4001d110e2f930c2e6163ccb2691cfe79869e8a: Status 404 returned error can't find the container with id 342573c5a2a5736a18b7a9b2a4001d110e2f930c2e6163ccb2691cfe79869e8a Sep 16 13:43:22.767648 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.767626 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-qsvk6"] Sep 16 13:43:22.768628 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:22.768610 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c00f5f_761c_4d82_a657_52e437aee43d.slice/crio-e1e899c6290de5c0c44615a4d2d57584c70d3ef4180cca63430122868d0c14b1 WatchSource:0}: Error finding container e1e899c6290de5c0c44615a4d2d57584c70d3ef4180cca63430122868d0c14b1: Status 404 returned error can't find the container with id e1e899c6290de5c0c44615a4d2d57584c70d3ef4180cca63430122868d0c14b1 Sep 16 13:43:22.769586 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:22.769510 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8736f18a_8bcd_4118_a84b_8d6526243746.slice/crio-71445049b5a8ae5b6cde77d07034ec2c026e0e3bd55837153f53a78e7d6294d8 WatchSource:0}: Error finding container 71445049b5a8ae5b6cde77d07034ec2c026e0e3bd55837153f53a78e7d6294d8: Status 404 returned error can't find the container with id 71445049b5a8ae5b6cde77d07034ec2c026e0e3bd55837153f53a78e7d6294d8 Sep 16 13:43:22.773630 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.773587 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-52kgm"] Sep 16 13:43:22.777555 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:22.777534 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8002868_fbfd_4166_9b6e_cb7e79f4d499.slice/crio-7a890475091916a62d458a6f22cd21bc6eece4a7a94489e00c707f2b34405e91 WatchSource:0}: Error finding container 7a890475091916a62d458a6f22cd21bc6eece4a7a94489e00c707f2b34405e91: Status 404 returned error can't find the container with id 7a890475091916a62d458a6f22cd21bc6eece4a7a94489e00c707f2b34405e91 Sep 16 13:43:22.787953 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.787929 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct"] Sep 16 13:43:22.791497 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.791472 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5brjf"] Sep 16 13:43:22.791671 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:22.791654 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5c390b_795a_4aab_b5b0_b4dad8c51074.slice/crio-e277a94fc2d03cb6fe7f277e6927a43d042adff9437beeb6dd446089a9937026 WatchSource:0}: Error finding container e277a94fc2d03cb6fe7f277e6927a43d042adff9437beeb6dd446089a9937026: Status 404 returned error can't find the container with id e277a94fc2d03cb6fe7f277e6927a43d042adff9437beeb6dd446089a9937026 Sep 16 13:43:22.792687 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:22.792664 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf"] Sep 16 13:43:22.795252 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:22.795230 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c5b25d_bb84_4667_b18e_30771d4c1c25.slice/crio-56231d270be0469177bdbb5ca5b43bd33b8f87a388d0ba456d789cea4ac979f1 WatchSource:0}: Error finding container 56231d270be0469177bdbb5ca5b43bd33b8f87a388d0ba456d789cea4ac979f1: Status 404 returned error can't find the container with id 56231d270be0469177bdbb5ca5b43bd33b8f87a388d0ba456d789cea4ac979f1 Sep 16 13:43:22.796778 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:22.796753 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9590e744_474c_4da6_a84f_eee73056b9c9.slice/crio-3f589d9342686825aa89302a03ba1811f356481eedee485a80aa5ffa041d4312 WatchSource:0}: Error finding container 3f589d9342686825aa89302a03ba1811f356481eedee485a80aa5ffa041d4312: Status 404 returned error can't find the container with id 3f589d9342686825aa89302a03ba1811f356481eedee485a80aa5ffa041d4312 Sep 16 13:43:23.674670 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:23.674589 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" event={"ID":"8736f18a-8bcd-4118-a84b-8d6526243746","Type":"ContainerStarted","Data":"71445049b5a8ae5b6cde77d07034ec2c026e0e3bd55837153f53a78e7d6294d8"} Sep 16 13:43:23.676678 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:23.676651 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" event={"ID":"80c00f5f-761c-4d82-a657-52e437aee43d","Type":"ContainerStarted","Data":"e1e899c6290de5c0c44615a4d2d57584c70d3ef4180cca63430122868d0c14b1"} Sep 16 13:43:23.678367 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:23.678328 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5brjf" event={"ID":"15c5b25d-bb84-4667-b18e-30771d4c1c25","Type":"ContainerStarted","Data":"56231d270be0469177bdbb5ca5b43bd33b8f87a388d0ba456d789cea4ac979f1"} Sep 16 13:43:23.680056 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:23.679895 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6nvrv" event={"ID":"97f4c753-e774-452c-8c8d-4070badd940e","Type":"ContainerStarted","Data":"342573c5a2a5736a18b7a9b2a4001d110e2f930c2e6163ccb2691cfe79869e8a"} Sep 16 13:43:23.682665 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:23.682639 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-52kgm" event={"ID":"d8002868-fbfd-4166-9b6e-cb7e79f4d499","Type":"ContainerStarted","Data":"7a890475091916a62d458a6f22cd21bc6eece4a7a94489e00c707f2b34405e91"} Sep 16 13:43:23.684314 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:23.684284 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" event={"ID":"9590e744-474c-4da6-a84f-eee73056b9c9","Type":"ContainerStarted","Data":"3f589d9342686825aa89302a03ba1811f356481eedee485a80aa5ffa041d4312"} Sep 16 13:43:23.691700 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:23.691223 2581 generic.go:356] "Generic (PLEG): container finished" podID="740f47ff-eb88-4272-8677-19e1d3c2148f" containerID="19979f32be6b0f4d0ad5e2b52826a8225277ed801c467622a6ce0f80b7c07369" exitCode=0 Sep 16 13:43:23.691700 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:23.691291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" event={"ID":"740f47ff-eb88-4272-8677-19e1d3c2148f","Type":"ContainerDied","Data":"19979f32be6b0f4d0ad5e2b52826a8225277ed801c467622a6ce0f80b7c07369"} Sep 16 13:43:23.695617 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:23.695358 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct" event={"ID":"9c5c390b-795a-4aab-b5b0-b4dad8c51074","Type":"ContainerStarted","Data":"e277a94fc2d03cb6fe7f277e6927a43d042adff9437beeb6dd446089a9937026"} Sep 16 13:43:24.248010 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:24.247975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:24.248252 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:24.248093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:24.248252 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:24.248156 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:24.248384 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.248291 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:43:24.248384 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.248306 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fcbbf884-59lgn: secret "image-registry-tls" not found Sep 16 13:43:24.248384 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.248365 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls podName:28371faa-0a08-49cf-8a63-9abe68aca3bc nodeName:}" failed. No retries permitted until 2026-09-16 13:43:28.248347875 +0000 UTC m=+42.211706703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls") pod "image-registry-67fcbbf884-59lgn" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc") : secret "image-registry-tls" not found Sep 16 13:43:24.248543 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.248429 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:24.248543 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.248461 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls podName:e2f131fb-a923-428d-82eb-39376dbd5594 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:28.248451027 +0000 UTC m=+42.211809847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gtcr" (UID: "e2f131fb-a923-428d-82eb-39376dbd5594") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:24.248543 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.248511 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:43:24.248543 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.248542 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls podName:de3aec25-00f6-4571-bedb-0e4f2a1dfdba nodeName:}" failed. No retries permitted until 2026-09-16 13:43:28.248530326 +0000 UTC m=+42.211889146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rnhz4" (UID: "de3aec25-00f6-4571-bedb-0e4f2a1dfdba") : secret "samples-operator-tls" not found Sep 16 13:43:24.348758 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:24.348714 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:24.348918 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:24.348835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:24.348989 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.348970 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:43:24.349044 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.349032 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:28.349014253 +0000 UTC m=+42.312373071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : secret "router-metrics-certs-default" not found Sep 16 13:43:24.349470 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.349451 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:28.349432793 +0000 UTC m=+42.312791597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : configmap references non-existent config key: service-ca.crt Sep 16 13:43:24.449538 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:24.449501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:24.449718 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.449666 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:43:24.449779 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:24.449726 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls podName:039c92e5-b92f-4090-867c-5ab2dd1ed94c nodeName:}" failed. No retries permitted until 2026-09-16 13:43:28.449708545 +0000 UTC m=+42.413067356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls") pod "dns-default-t2dsl" (UID: "039c92e5-b92f-4090-867c-5ab2dd1ed94c") : secret "dns-default-metrics-tls" not found Sep 16 13:43:24.708901 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:24.708663 2581 generic.go:356] "Generic (PLEG): container finished" podID="740f47ff-eb88-4272-8677-19e1d3c2148f" containerID="ba0e7ac9a936065d62d950aa479b2d5a3f15d93ace2a05f8a0315b5830a95eb0" exitCode=0 Sep 16 13:43:24.708901 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:24.708717 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" event={"ID":"740f47ff-eb88-4272-8677-19e1d3c2148f","Type":"ContainerDied","Data":"ba0e7ac9a936065d62d950aa479b2d5a3f15d93ace2a05f8a0315b5830a95eb0"} Sep 16 13:43:25.719436 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:25.719397 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" event={"ID":"740f47ff-eb88-4272-8677-19e1d3c2148f","Type":"ContainerStarted","Data":"9091d10d0fa3bf7e6ab830ec9b7ec842a340af4843106937e6a58f7765f7ec26"} Sep 16 13:43:25.746009 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:25.745951 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4pwdh" podStartSLOduration=7.658374904 podStartE2EDuration="38.745935026s" totalImagesPullingTime="31.087560122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:25.745004226 +0000 UTC m=+39.708363066" watchObservedRunningTime="2026-09-16 13:43:25.745935026 +0000 UTC m=+39.709293852" Sep 16 13:43:28.290233 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:28.290198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:28.290635 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:28.290282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:28.290635 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:28.290375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:28.290635 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.290379 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:43:28.290635 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.290402 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fcbbf884-59lgn: secret "image-registry-tls" not found Sep 16 13:43:28.290635 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.290428 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:28.290635 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.290474 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls podName:28371faa-0a08-49cf-8a63-9abe68aca3bc nodeName:}" failed. No retries permitted until 2026-09-16 13:43:36.290454359 +0000 UTC m=+50.253813184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls") pod "image-registry-67fcbbf884-59lgn" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc") : secret "image-registry-tls" not found Sep 16 13:43:28.290635 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.290488 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls podName:e2f131fb-a923-428d-82eb-39376dbd5594 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:36.290482272 +0000 UTC m=+50.253841077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gtcr" (UID: "e2f131fb-a923-428d-82eb-39376dbd5594") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:28.290635 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.290488 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:43:28.290635 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.290543 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls podName:de3aec25-00f6-4571-bedb-0e4f2a1dfdba nodeName:}" failed. No retries permitted until 2026-09-16 13:43:36.290527739 +0000 UTC m=+50.253886546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rnhz4" (UID: "de3aec25-00f6-4571-bedb-0e4f2a1dfdba") : secret "samples-operator-tls" not found Sep 16 13:43:28.391158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:28.391116 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:28.391308 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:28.391221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:28.391308 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.391271 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:36.391255186 +0000 UTC m=+50.354613991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : configmap references non-existent config key: service-ca.crt Sep 16 13:43:28.391432 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.391334 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:43:28.391432 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.391409 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:36.391393247 +0000 UTC m=+50.354752076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : secret "router-metrics-certs-default" not found Sep 16 13:43:28.492108 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:28.492072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:28.492291 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.492225 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:43:28.492358 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:28.492306 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls podName:039c92e5-b92f-4090-867c-5ab2dd1ed94c nodeName:}" failed. No retries permitted until 2026-09-16 13:43:36.492285848 +0000 UTC m=+50.455644674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls") pod "dns-default-t2dsl" (UID: "039c92e5-b92f-4090-867c-5ab2dd1ed94c") : secret "dns-default-metrics-tls" not found Sep 16 13:43:30.736453 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.736079 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct" event={"ID":"9c5c390b-795a-4aab-b5b0-b4dad8c51074","Type":"ContainerStarted","Data":"11296524adefeb6e89b46a2d62f2b5125a9471a7fe85fb1850e32f7a23b0fc4c"} Sep 16 13:43:30.738133 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.738112 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/0.log" Sep 16 13:43:30.738264 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.738169 2581 generic.go:356] "Generic (PLEG): container finished" podID="8736f18a-8bcd-4118-a84b-8d6526243746" containerID="45d7b2fc7b2061f248013e2b71594c3ec9c28624f3de371807870e5e754ac8b1" exitCode=255 Sep 16 13:43:30.738396 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.738378 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" event={"ID":"8736f18a-8bcd-4118-a84b-8d6526243746","Type":"ContainerDied","Data":"45d7b2fc7b2061f248013e2b71594c3ec9c28624f3de371807870e5e754ac8b1"} Sep 16 13:43:30.739274 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.739257 2581 scope.go:117] "RemoveContainer" containerID="45d7b2fc7b2061f248013e2b71594c3ec9c28624f3de371807870e5e754ac8b1" Sep 16 13:43:30.742646 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.742622 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" event={"ID":"80c00f5f-761c-4d82-a657-52e437aee43d","Type":"ContainerStarted","Data":"f599d02909da616216c9d109e389c485370289c30b1e868a58744c2c9fc759c8"} Sep 16 13:43:30.745772 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.745455 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5brjf" event={"ID":"15c5b25d-bb84-4667-b18e-30771d4c1c25","Type":"ContainerStarted","Data":"7d3cf4e9deaa2b597d30f60cbd6ca8253885999faf14b7d61078242b7dbaa93a"} Sep 16 13:43:30.747029 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.746976 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6nvrv" event={"ID":"97f4c753-e774-452c-8c8d-4070badd940e","Type":"ContainerStarted","Data":"dcf9201a55b5f3e277baba70b09774ccf099058e3d3b78d5705f43737eb68447"} Sep 16 13:43:30.748494 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.748452 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-52kgm" event={"ID":"d8002868-fbfd-4166-9b6e-cb7e79f4d499","Type":"ContainerStarted","Data":"fc0021034303ed373ee8a7ed880f7f7dfb07481c21be6d421a47b9c9de9cd5b7"} Sep 16 13:43:30.750136 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.750113 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" event={"ID":"9590e744-474c-4da6-a84f-eee73056b9c9","Type":"ContainerStarted","Data":"26ea916afc9c2c59d27ce2824ebb50c964555df95cee8237cec65a28e2ff9f6f"} Sep 16 13:43:30.760834 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.760608 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-hslct" podStartSLOduration=19.186750962 podStartE2EDuration="26.760589289s" totalImagesPullingTime="7.573838327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:30.759605509 +0000 UTC m=+44.722964337" watchObservedRunningTime="2026-09-16 13:43:30.760589289 +0000 UTC m=+44.723948117" Sep 16 13:43:30.808926 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.808875 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" podStartSLOduration=15.087823326 podStartE2EDuration="22.808856443s" totalImagesPullingTime="7.721033117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:30.780058044 +0000 UTC m=+44.743416873" watchObservedRunningTime="2026-09-16 13:43:30.808856443 +0000 UTC m=+44.772215267" Sep 16 13:43:30.809369 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.809326 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5brjf" podStartSLOduration=3.114918169 podStartE2EDuration="10.809314703s" totalImagesPullingTime="7.694396534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:30.80866252 +0000 UTC m=+44.772021351" watchObservedRunningTime="2026-09-16 13:43:30.809314703 +0000 UTC m=+44.772673530" Sep 16 13:43:30.831393 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.831347 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-52kgm" podStartSLOduration=14.109348934 podStartE2EDuration="21.831328478s" totalImagesPullingTime="7.721979544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:30.829795118 +0000 UTC m=+44.793153950" watchObservedRunningTime="2026-09-16 13:43:30.831328478 +0000 UTC m=+44.794687301" Sep 16 13:43:30.890272 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.890215 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-6nvrv" podStartSLOduration=19.166145346 podStartE2EDuration="26.890193816s" totalImagesPullingTime="7.72404847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:30.858065157 +0000 UTC m=+44.821423985" watchObservedRunningTime="2026-09-16 13:43:30.890193816 +0000 UTC m=+44.853552643" Sep 16 13:43:30.917693 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.917308 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:30.917693 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.917430 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:30.926366 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:30.926310 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" podStartSLOduration=17.23280906 podStartE2EDuration="24.926289815s" totalImagesPullingTime="7.693480755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:30.924575399 +0000 UTC m=+44.887934229" watchObservedRunningTime="2026-09-16 13:43:30.926289815 +0000 UTC m=+44.889648644" Sep 16 13:43:31.755214 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.754910 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 13:43:31.757508 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.755909 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/0.log" Sep 16 13:43:31.757508 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.755947 2581 generic.go:356] "Generic (PLEG): container finished" podID="8736f18a-8bcd-4118-a84b-8d6526243746" containerID="0742d96ec3fcee3b65646c835262cb42e0ed2d81b01dc756c1287e927011435f" exitCode=255 Sep 16 13:43:31.757508 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.756700 2581 scope.go:117] "RemoveContainer" containerID="0742d96ec3fcee3b65646c835262cb42e0ed2d81b01dc756c1287e927011435f" Sep 16 13:43:31.757508 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:31.756845 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-qsvk6_openshift-console-operator(8736f18a-8bcd-4118-a84b-8d6526243746)\"" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" podUID="8736f18a-8bcd-4118-a84b-8d6526243746" Sep 16 13:43:31.757508 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.757239 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" event={"ID":"8736f18a-8bcd-4118-a84b-8d6526243746","Type":"ContainerDied","Data":"0742d96ec3fcee3b65646c835262cb42e0ed2d81b01dc756c1287e927011435f"} Sep 16 13:43:31.757508 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.757292 2581 scope.go:117] "RemoveContainer" containerID="45d7b2fc7b2061f248013e2b71594c3ec9c28624f3de371807870e5e754ac8b1" Sep 16 13:43:31.816403 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.816373 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w7g9k"] Sep 16 13:43:31.839318 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.839285 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w7g9k"] Sep 16 13:43:31.839467 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.839444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:31.842682 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.842660 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 13:43:31.842831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.842660 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 13:43:31.843021 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.843005 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75dks\"" Sep 16 13:43:31.921796 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.921753 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:31.921973 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.921854 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfz2c\" (UniqueName: \"kubernetes.io/projected/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-api-access-cfz2c\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:31.921973 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.921907 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:31.922071 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.921998 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d7a8d13-6484-443a-805b-061bbfdb8e19-data-volume\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:31.922071 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:31.922031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d7a8d13-6484-443a-805b-061bbfdb8e19-crio-socket\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.022876 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfz2c\" (UniqueName: \"kubernetes.io/projected/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-api-access-cfz2c\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.022943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.023012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d7a8d13-6484-443a-805b-061bbfdb8e19-data-volume\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.023043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d7a8d13-6484-443a-805b-061bbfdb8e19-crio-socket\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.023123 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:32.023269 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:32.023327 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls podName:3d7a8d13-6484-443a-805b-061bbfdb8e19 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:32.523309788 +0000 UTC m=+46.486668613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7g9k" (UID: "3d7a8d13-6484-443a-805b-061bbfdb8e19") : secret "insights-runtime-extractor-tls" not found Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.023572 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d7a8d13-6484-443a-805b-061bbfdb8e19-data-volume\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:32.023647 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.023660 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d7a8d13-6484-443a-805b-061bbfdb8e19-crio-socket\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.023712 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:32.023700 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm podName:3d7a8d13-6484-443a-805b-061bbfdb8e19 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:32.523681894 +0000 UTC m=+46.487040714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7g9k" (UID: "3d7a8d13-6484-443a-805b-061bbfdb8e19") : configmap "kube-rbac-proxy" not found Sep 16 13:43:32.043531 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.043504 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfz2c\" (UniqueName: \"kubernetes.io/projected/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-api-access-cfz2c\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.526290 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.526260 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.526446 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.526329 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:32.526446 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:32.526417 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:43:32.526519 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:32.526445 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:43:32.526519 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:32.526490 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls podName:3d7a8d13-6484-443a-805b-061bbfdb8e19 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:33.526469995 +0000 UTC m=+47.489828800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7g9k" (UID: "3d7a8d13-6484-443a-805b-061bbfdb8e19") : secret "insights-runtime-extractor-tls" not found Sep 16 13:43:32.526519 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:32.526510 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm podName:3d7a8d13-6484-443a-805b-061bbfdb8e19 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:33.526499649 +0000 UTC m=+47.489858460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7g9k" (UID: "3d7a8d13-6484-443a-805b-061bbfdb8e19") : configmap "kube-rbac-proxy" not found Sep 16 13:43:32.765885 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.765855 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 13:43:32.766270 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:32.766201 2581 scope.go:117] "RemoveContainer" containerID="0742d96ec3fcee3b65646c835262cb42e0ed2d81b01dc756c1287e927011435f" Sep 16 13:43:32.766359 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:32.766340 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-qsvk6_openshift-console-operator(8736f18a-8bcd-4118-a84b-8d6526243746)\"" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" podUID="8736f18a-8bcd-4118-a84b-8d6526243746" Sep 16 13:43:33.182154 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:33.182108 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gvnzs_cd6779bd-b278-4b06-a7ca-80a4aae73ae0/dns-node-resolver/0.log" Sep 16 13:43:33.534073 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:33.534035 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:33.534295 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:33.534209 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:43:33.534295 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:33.534265 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:43:33.534295 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:33.534284 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls podName:3d7a8d13-6484-443a-805b-061bbfdb8e19 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:35.534268597 +0000 UTC m=+49.497627401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7g9k" (UID: "3d7a8d13-6484-443a-805b-061bbfdb8e19") : secret "insights-runtime-extractor-tls" not found Sep 16 13:43:33.534444 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:33.534334 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm podName:3d7a8d13-6484-443a-805b-061bbfdb8e19 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:35.534318625 +0000 UTC m=+49.497677453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7g9k" (UID: "3d7a8d13-6484-443a-805b-061bbfdb8e19") : configmap "kube-rbac-proxy" not found Sep 16 13:43:33.534444 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:33.534208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:33.769040 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:33.769014 2581 scope.go:117] "RemoveContainer" containerID="0742d96ec3fcee3b65646c835262cb42e0ed2d81b01dc756c1287e927011435f" Sep 16 13:43:33.769412 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:33.769191 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-qsvk6_openshift-console-operator(8736f18a-8bcd-4118-a84b-8d6526243746)\"" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" podUID="8736f18a-8bcd-4118-a84b-8d6526243746" Sep 16 13:43:34.385358 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:34.385327 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n97z8_f4305a9e-2784-4ecd-9462-64b45a220538/node-ca/0.log" Sep 16 13:43:34.986759 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:34.986726 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5brjf_15c5b25d-bb84-4667-b18e-30771d4c1c25/serve-healthcheck-canary/0.log" Sep 16 13:43:35.551645 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:35.551613 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:35.551813 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:35.551685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:35.551813 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:35.551763 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:43:35.551884 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:35.551809 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:43:35.551884 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:35.551831 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls podName:3d7a8d13-6484-443a-805b-061bbfdb8e19 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:39.551816267 +0000 UTC m=+53.515175072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7g9k" (UID: "3d7a8d13-6484-443a-805b-061bbfdb8e19") : secret "insights-runtime-extractor-tls" not found Sep 16 13:43:35.551884 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:35.551865 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm podName:3d7a8d13-6484-443a-805b-061bbfdb8e19 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:39.551853292 +0000 UTC m=+53.515212103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7g9k" (UID: "3d7a8d13-6484-443a-805b-061bbfdb8e19") : configmap "kube-rbac-proxy" not found Sep 16 13:43:36.359861 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:36.359831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:36.360250 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:36.359882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:36.360250 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:36.359941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:36.360250 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.360008 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:43:36.360250 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.360036 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:36.360250 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.360039 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:43:36.360250 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.360055 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fcbbf884-59lgn: secret "image-registry-tls" not found Sep 16 13:43:36.360250 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.360089 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls podName:de3aec25-00f6-4571-bedb-0e4f2a1dfdba nodeName:}" failed. No retries permitted until 2026-09-16 13:43:52.360070826 +0000 UTC m=+66.323429647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rnhz4" (UID: "de3aec25-00f6-4571-bedb-0e4f2a1dfdba") : secret "samples-operator-tls" not found Sep 16 13:43:36.360250 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.360106 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls podName:28371faa-0a08-49cf-8a63-9abe68aca3bc nodeName:}" failed. No retries permitted until 2026-09-16 13:43:52.360098714 +0000 UTC m=+66.323457520 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls") pod "image-registry-67fcbbf884-59lgn" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc") : secret "image-registry-tls" not found Sep 16 13:43:36.360250 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.360120 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls podName:e2f131fb-a923-428d-82eb-39376dbd5594 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:52.360113611 +0000 UTC m=+66.323472415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-8gtcr" (UID: "e2f131fb-a923-428d-82eb-39376dbd5594") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:43:36.461260 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:36.461225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:36.461439 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:36.461408 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:36.461504 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.461423 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:43:36.461552 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.461536 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:52.461521078 +0000 UTC m=+66.424879901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : configmap references non-existent config key: service-ca.crt Sep 16 13:43:36.461602 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.461561 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs podName:5520568f-3a4f-4665-b75c-d4726ccfebb1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:52.461551094 +0000 UTC m=+66.424909913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs") pod "router-default-bcdcbb9b-kbf6r" (UID: "5520568f-3a4f-4665-b75c-d4726ccfebb1") : secret "router-metrics-certs-default" not found Sep 16 13:43:36.561915 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:36.561887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:36.562047 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.562033 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:43:36.562102 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:36.562091 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls podName:039c92e5-b92f-4090-867c-5ab2dd1ed94c nodeName:}" failed. No retries permitted until 2026-09-16 13:43:52.562078106 +0000 UTC m=+66.525436917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls") pod "dns-default-t2dsl" (UID: "039c92e5-b92f-4090-867c-5ab2dd1ed94c") : secret "dns-default-metrics-tls" not found Sep 16 13:43:39.586597 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:39.586559 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:39.586989 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:39.586638 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:39.586989 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:39.586719 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:43:39.586989 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:39.586793 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls podName:3d7a8d13-6484-443a-805b-061bbfdb8e19 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:47.58677344 +0000 UTC m=+61.550132250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7g9k" (UID: "3d7a8d13-6484-443a-805b-061bbfdb8e19") : secret "insights-runtime-extractor-tls" not found Sep 16 13:43:39.587327 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:39.587309 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d7a8d13-6484-443a-805b-061bbfdb8e19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:40.917812 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:40.917782 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:40.918240 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:40.917836 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:40.918240 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:40.918131 2581 scope.go:117] "RemoveContainer" containerID="0742d96ec3fcee3b65646c835262cb42e0ed2d81b01dc756c1287e927011435f" Sep 16 13:43:40.918324 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:40.918296 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-qsvk6_openshift-console-operator(8736f18a-8bcd-4118-a84b-8d6526243746)\"" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" podUID="8736f18a-8bcd-4118-a84b-8d6526243746" Sep 16 13:43:45.659112 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:45.659082 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g58wg" Sep 16 13:43:47.645953 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:47.645915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:47.648241 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:47.648217 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d7a8d13-6484-443a-805b-061bbfdb8e19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7g9k\" (UID: \"3d7a8d13-6484-443a-805b-061bbfdb8e19\") " pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:47.753937 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:47.753905 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75dks\"" Sep 16 13:43:47.761567 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:47.761544 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w7g9k" Sep 16 13:43:48.047448 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:48.047419 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w7g9k"] Sep 16 13:43:48.053610 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:48.053582 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7a8d13_6484_443a_805b_061bbfdb8e19.slice/crio-283b072d2c8455868202523987fedaefbd409202f562af86665beeb272e6491e WatchSource:0}: Error finding container 283b072d2c8455868202523987fedaefbd409202f562af86665beeb272e6491e: Status 404 returned error can't find the container with id 283b072d2c8455868202523987fedaefbd409202f562af86665beeb272e6491e Sep 16 13:43:48.816968 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:48.816940 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7g9k" event={"ID":"3d7a8d13-6484-443a-805b-061bbfdb8e19","Type":"ContainerStarted","Data":"d9aa8d0f7b359678705815b275082f0cf9a850f1ab18e668df47cfeb6b136f2a"} Sep 16 13:43:48.817256 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:48.816995 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7g9k" event={"ID":"3d7a8d13-6484-443a-805b-061bbfdb8e19","Type":"ContainerStarted","Data":"d141795a5dae606ee68c557e2129c27052f5c2ca079ff7216edafc474dfc9ef7"} Sep 16 13:43:48.817256 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:48.817015 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7g9k" event={"ID":"3d7a8d13-6484-443a-805b-061bbfdb8e19","Type":"ContainerStarted","Data":"283b072d2c8455868202523987fedaefbd409202f562af86665beeb272e6491e"} Sep 16 13:43:50.825552 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:50.825517 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7g9k" event={"ID":"3d7a8d13-6484-443a-805b-061bbfdb8e19","Type":"ContainerStarted","Data":"fe71af4aab929e5d16fc9cc568ba40d029e01e284be8f2f73eb7f84333f91511"} Sep 16 13:43:50.847489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:50.847445 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w7g9k" podStartSLOduration=17.806340286 podStartE2EDuration="19.847431016s" totalImagesPullingTime="2.04109073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:50.846810654 +0000 UTC m=+64.810169493" watchObservedRunningTime="2026-09-16 13:43:50.847431016 +0000 UTC m=+64.810789843" Sep 16 13:43:52.282188 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.282125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:52.285174 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.285137 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 13:43:52.294717 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.294694 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42514386-d0ca-4a5b-b5aa-d0e5b3276cd0-metrics-certs\") pod \"network-metrics-daemon-9dzbs\" (UID: \"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0\") " pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:52.328110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.328085 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7xrvw\"" Sep 16 13:43:52.335748 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.335732 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dzbs" Sep 16 13:43:52.411973 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.411930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:52.412104 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.412050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:52.412104 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.412096 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:52.412242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.412171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:52.412242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.412204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696t4\" (UniqueName: \"kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4\") pod \"network-check-target-mcl5r\" (UID: \"f0d7ecc6-7b55-4cca-9c55-093d59c1e854\") " pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:52.415204 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.415184 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-696t4\" (UniqueName: \"kubernetes.io/projected/f0d7ecc6-7b55-4cca-9c55-093d59c1e854-kube-api-access-696t4\") pod \"network-check-target-mcl5r\" (UID: \"f0d7ecc6-7b55-4cca-9c55-093d59c1e854\") " pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:52.415601 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.415579 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2f131fb-a923-428d-82eb-39376dbd5594-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-8gtcr\" (UID: \"e2f131fb-a923-428d-82eb-39376dbd5594\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:52.416004 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.415988 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 13:43:52.416495 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.416477 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de3aec25-00f6-4571-bedb-0e4f2a1dfdba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rnhz4\" (UID: \"de3aec25-00f6-4571-bedb-0e4f2a1dfdba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:52.416793 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.416779 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") pod \"image-registry-67fcbbf884-59lgn\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:52.425202 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.425182 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1499dfa1-b0ce-44a0-add8-c1404440ecdd-original-pull-secret\") pod \"global-pull-secret-syncer-hddml\" (UID: \"1499dfa1-b0ce-44a0-add8-c1404440ecdd\") " pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:52.512868 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.512839 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:52.512974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.512897 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:52.513407 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.513389 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5520568f-3a4f-4665-b75c-d4726ccfebb1-service-ca-bundle\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:52.514816 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.514796 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5520568f-3a4f-4665-b75c-d4726ccfebb1-metrics-certs\") pod \"router-default-bcdcbb9b-kbf6r\" (UID: \"5520568f-3a4f-4665-b75c-d4726ccfebb1\") " pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:52.612814 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.612757 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9dzbs"] Sep 16 13:43:52.613066 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.613051 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-97tnf\"" Sep 16 13:43:52.613196 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.613178 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:52.615091 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.615073 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/039c92e5-b92f-4090-867c-5ab2dd1ed94c-metrics-tls\") pod \"dns-default-t2dsl\" (UID: \"039c92e5-b92f-4090-867c-5ab2dd1ed94c\") " pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:52.615868 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:52.615849 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42514386_d0ca_4a5b_b5aa_d0e5b3276cd0.slice/crio-c3b3c3ba9ed9ba14f3b9a1e95985979833cb50306599c94ca48fc1cff3af990e WatchSource:0}: Error finding container c3b3c3ba9ed9ba14f3b9a1e95985979833cb50306599c94ca48fc1cff3af990e: Status 404 returned error can't find the container with id c3b3c3ba9ed9ba14f3b9a1e95985979833cb50306599c94ca48fc1cff3af990e Sep 16 13:43:52.620120 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.620105 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" Sep 16 13:43:52.635557 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.635538 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4knh2\"" Sep 16 13:43:52.643270 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.643228 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:52.643523 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.643508 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s4xvt\"" Sep 16 13:43:52.648447 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.648429 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hddml" Sep 16 13:43:52.651059 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.651039 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:52.657010 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.656989 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zs72k\"" Sep 16 13:43:52.663579 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.663549 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" Sep 16 13:43:52.705976 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.705947 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6szcg\"" Sep 16 13:43:52.713860 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.713466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:52.759205 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.759180 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kqx7z\"" Sep 16 13:43:52.766818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.766797 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:52.835127 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.835099 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dzbs" event={"ID":"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0","Type":"ContainerStarted","Data":"c3b3c3ba9ed9ba14f3b9a1e95985979833cb50306599c94ca48fc1cff3af990e"} Sep 16 13:43:52.908590 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.908569 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4"] Sep 16 13:43:52.945388 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.945063 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mcl5r"] Sep 16 13:43:52.947102 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:52.947080 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d7ecc6_7b55_4cca_9c55_093d59c1e854.slice/crio-bc3de2cbff9cb895b87ec456e403592fb2b8593473c74fc5f75ec1e78fb5ba7f WatchSource:0}: Error finding container bc3de2cbff9cb895b87ec456e403592fb2b8593473c74fc5f75ec1e78fb5ba7f: Status 404 returned error can't find the container with id bc3de2cbff9cb895b87ec456e403592fb2b8593473c74fc5f75ec1e78fb5ba7f Sep 16 13:43:52.965690 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.965520 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67fcbbf884-59lgn"] Sep 16 13:43:52.976961 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:52.976941 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28371faa_0a08_49cf_8a63_9abe68aca3bc.slice/crio-863e9bd8b23adec72c914b1461a79c24f8639a5c032350dff1663bc1ad8a6725 WatchSource:0}: Error finding container 863e9bd8b23adec72c914b1461a79c24f8639a5c032350dff1663bc1ad8a6725: Status 404 returned error can't find the container with id 863e9bd8b23adec72c914b1461a79c24f8639a5c032350dff1663bc1ad8a6725 Sep 16 13:43:52.986806 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.986734 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hddml"] Sep 16 13:43:52.988535 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:52.988480 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr"] Sep 16 13:43:52.992891 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:52.992853 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1499dfa1_b0ce_44a0_add8_c1404440ecdd.slice/crio-a5d3c16f7771e2fe71533bc7566b588dc0ca29ff807a4ef5c97656752611d39c WatchSource:0}: Error finding container a5d3c16f7771e2fe71533bc7566b588dc0ca29ff807a4ef5c97656752611d39c: Status 404 returned error can't find the container with id a5d3c16f7771e2fe71533bc7566b588dc0ca29ff807a4ef5c97656752611d39c Sep 16 13:43:52.993539 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:52.993484 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f131fb_a923_428d_82eb_39376dbd5594.slice/crio-ffc6726ada3c362a5415314fddce537c535851ecddd698d4311f13b69117d906 WatchSource:0}: Error finding container ffc6726ada3c362a5415314fddce537c535851ecddd698d4311f13b69117d906: Status 404 returned error can't find the container with id ffc6726ada3c362a5415314fddce537c535851ecddd698d4311f13b69117d906 Sep 16 13:43:53.026911 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.026882 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-bcdcbb9b-kbf6r"] Sep 16 13:43:53.029198 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:53.029170 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5520568f_3a4f_4665_b75c_d4726ccfebb1.slice/crio-975a64fda302a9679e4cbe7ed89eaad252903bdc69f878e75a4ceb1d85354940 WatchSource:0}: Error finding container 975a64fda302a9679e4cbe7ed89eaad252903bdc69f878e75a4ceb1d85354940: Status 404 returned error can't find the container with id 975a64fda302a9679e4cbe7ed89eaad252903bdc69f878e75a4ceb1d85354940 Sep 16 13:43:53.052656 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.052629 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2dsl"] Sep 16 13:43:53.055465 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:43:53.055443 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039c92e5_b92f_4090_867c_5ab2dd1ed94c.slice/crio-350ada39bc6b9e30649b0b6a3c14740388834d94fda08bfe91899a1c627017b6 WatchSource:0}: Error finding container 350ada39bc6b9e30649b0b6a3c14740388834d94fda08bfe91899a1c627017b6: Status 404 returned error can't find the container with id 350ada39bc6b9e30649b0b6a3c14740388834d94fda08bfe91899a1c627017b6 Sep 16 13:43:53.846979 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.846129 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" event={"ID":"5520568f-3a4f-4665-b75c-d4726ccfebb1","Type":"ContainerStarted","Data":"f1c8e5570076dc7bcf65426b71572a8e43c06eb343100909de4c3e4fc604aafc"} Sep 16 13:43:53.846979 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.846203 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" event={"ID":"5520568f-3a4f-4665-b75c-d4726ccfebb1","Type":"ContainerStarted","Data":"975a64fda302a9679e4cbe7ed89eaad252903bdc69f878e75a4ceb1d85354940"} Sep 16 13:43:53.849715 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.849660 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2dsl" event={"ID":"039c92e5-b92f-4090-867c-5ab2dd1ed94c","Type":"ContainerStarted","Data":"350ada39bc6b9e30649b0b6a3c14740388834d94fda08bfe91899a1c627017b6"} Sep 16 13:43:53.852298 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.852223 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" event={"ID":"e2f131fb-a923-428d-82eb-39376dbd5594","Type":"ContainerStarted","Data":"ffc6726ada3c362a5415314fddce537c535851ecddd698d4311f13b69117d906"} Sep 16 13:43:53.857222 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.856631 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" event={"ID":"28371faa-0a08-49cf-8a63-9abe68aca3bc","Type":"ContainerStarted","Data":"5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0"} Sep 16 13:43:53.857222 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.856672 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" event={"ID":"28371faa-0a08-49cf-8a63-9abe68aca3bc","Type":"ContainerStarted","Data":"863e9bd8b23adec72c914b1461a79c24f8639a5c032350dff1663bc1ad8a6725"} Sep 16 13:43:53.857222 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.857170 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:43:53.859012 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.858950 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" event={"ID":"de3aec25-00f6-4571-bedb-0e4f2a1dfdba","Type":"ContainerStarted","Data":"4104ee13984d6853866e11ec11bcb26e11e72bc141243e2fb5f5d95efa8898d6"} Sep 16 13:43:53.861008 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.860985 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hddml" event={"ID":"1499dfa1-b0ce-44a0-add8-c1404440ecdd","Type":"ContainerStarted","Data":"a5d3c16f7771e2fe71533bc7566b588dc0ca29ff807a4ef5c97656752611d39c"} Sep 16 13:43:53.865479 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.865452 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mcl5r" event={"ID":"f0d7ecc6-7b55-4cca-9c55-093d59c1e854","Type":"ContainerStarted","Data":"09e0b0c37e64ff3cc8223e7efc572a443f40b1c3d630475473e91bed4b4c155e"} Sep 16 13:43:53.865576 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.865499 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mcl5r" event={"ID":"f0d7ecc6-7b55-4cca-9c55-093d59c1e854","Type":"ContainerStarted","Data":"bc3de2cbff9cb895b87ec456e403592fb2b8593473c74fc5f75ec1e78fb5ba7f"} Sep 16 13:43:53.865916 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.865896 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:43:53.885489 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.885439 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" podStartSLOduration=49.885420139 podStartE2EDuration="49.885420139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:53.883157353 +0000 UTC m=+67.846516183" watchObservedRunningTime="2026-09-16 13:43:53.885420139 +0000 UTC m=+67.848778967" Sep 16 13:43:53.936339 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.936290 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mcl5r" podStartSLOduration=66.936270486 podStartE2EDuration="1m6.936270486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:53.912646824 +0000 UTC m=+67.876005653" watchObservedRunningTime="2026-09-16 13:43:53.936270486 +0000 UTC m=+67.899629314" Sep 16 13:43:53.936473 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:53.936389 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" podStartSLOduration=49.936381128 podStartE2EDuration="49.936381128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:53.935435311 +0000 UTC m=+67.898794139" watchObservedRunningTime="2026-09-16 13:43:53.936381128 +0000 UTC m=+67.899739956" Sep 16 13:43:54.714608 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:54.714571 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:54.718010 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:54.717987 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:54.872248 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:54.872203 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dzbs" event={"ID":"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0","Type":"ContainerStarted","Data":"83704e312d7780f40a298fe6df0aa461f9e5219c844ccb912c950ca1235e005a"} Sep 16 13:43:54.872656 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:54.872269 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dzbs" event={"ID":"42514386-d0ca-4a5b-b5aa-d0e5b3276cd0","Type":"ContainerStarted","Data":"b96e0dcd738dbe6a93054eaa5000d435009d616174687a09ab0119777812033f"} Sep 16 13:43:54.873200 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:54.873175 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:54.874927 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:54.874901 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-bcdcbb9b-kbf6r" Sep 16 13:43:54.891764 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:54.891631 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9dzbs" podStartSLOduration=66.415533504 podStartE2EDuration="1m7.891615831s" totalImagesPullingTime="1.476082327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:54.891073155 +0000 UTC m=+68.854431997" watchObservedRunningTime="2026-09-16 13:43:54.891615831 +0000 UTC m=+68.854974658" Sep 16 13:43:55.509780 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:55.509747 2581 scope.go:117] "RemoveContainer" containerID="0742d96ec3fcee3b65646c835262cb42e0ed2d81b01dc756c1287e927011435f" Sep 16 13:43:57.016919 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:57.016884 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67fcbbf884-59lgn"] Sep 16 13:43:58.893991 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.893947 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2dsl" event={"ID":"039c92e5-b92f-4090-867c-5ab2dd1ed94c","Type":"ContainerStarted","Data":"217fe0d3e059aec8e2c763d5ebd8f238b616a9b76e867d1d39ba7d2abee63869"} Sep 16 13:43:58.894544 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.894016 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2dsl" event={"ID":"039c92e5-b92f-4090-867c-5ab2dd1ed94c","Type":"ContainerStarted","Data":"30fb1b560cfbef43c8f2e218e0e23ff357050ea17d4b84b1ad7df66af5814599"} Sep 16 13:43:58.894544 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.894380 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t2dsl" Sep 16 13:43:58.896274 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.896245 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" event={"ID":"e2f131fb-a923-428d-82eb-39376dbd5594","Type":"ContainerStarted","Data":"a5d72c5949bc004ca2bc91016e289a06824541823baf6306694326b17bc15e09"} Sep 16 13:43:58.899280 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.898840 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" event={"ID":"de3aec25-00f6-4571-bedb-0e4f2a1dfdba","Type":"ContainerStarted","Data":"f9debe1feeb795ceb8f68c69d53b0a7e55142d35322c9ae503c31f85228be90d"} Sep 16 13:43:58.899280 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.898892 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" event={"ID":"de3aec25-00f6-4571-bedb-0e4f2a1dfdba","Type":"ContainerStarted","Data":"597bd445307b10167ce1643acf23845e31e0c3d4930bf54ac28883e69dbc3989"} Sep 16 13:43:58.901009 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.900984 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 13:43:58.901116 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.901095 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" event={"ID":"8736f18a-8bcd-4118-a84b-8d6526243746","Type":"ContainerStarted","Data":"e85df788915339248b33f4ad9dbc281a4845e3011f89b1b7a5402fe201f9c268"} Sep 16 13:43:58.901426 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.901406 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:58.902844 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.902822 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hddml" event={"ID":"1499dfa1-b0ce-44a0-add8-c1404440ecdd","Type":"ContainerStarted","Data":"8b82910d2a7ce76eed3dde216d07c53e801ff54b4e5187af090b204bda9e11b1"} Sep 16 13:43:58.922922 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.922881 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t2dsl" podStartSLOduration=33.796590949 podStartE2EDuration="38.922867144s" totalImagesPullingTime="5.126276195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:58.922237289 +0000 UTC m=+72.885596118" watchObservedRunningTime="2026-09-16 13:43:58.922867144 +0000 UTC m=+72.886226016" Sep 16 13:43:58.957553 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.957505 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hddml" podStartSLOduration=66.560823093 podStartE2EDuration="1m11.957490835s" totalImagesPullingTime="5.396667742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:58.956420342 +0000 UTC m=+72.919779170" watchObservedRunningTime="2026-09-16 13:43:58.957490835 +0000 UTC m=+72.920849662" Sep 16 13:43:58.983211 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.983152 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rnhz4" podStartSLOduration=49.74847571 podStartE2EDuration="54.98312666s" totalImagesPullingTime="5.23465095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:58.982221993 +0000 UTC m=+72.945580821" watchObservedRunningTime="2026-09-16 13:43:58.98312666 +0000 UTC m=+72.946485486" Sep 16 13:43:58.997748 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:58.997720 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt"] Sep 16 13:43:59.014350 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.014326 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" Sep 16 13:43:59.017328 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.017308 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 16 13:43:59.019021 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.018998 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-r4kt8\"" Sep 16 13:43:59.019453 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.019432 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt"] Sep 16 13:43:59.024405 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.024363 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-8gtcr" podStartSLOduration=49.83392569 podStartE2EDuration="55.02434744s" totalImagesPullingTime="5.19042175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:59.022254984 +0000 UTC m=+72.985613824" watchObservedRunningTime="2026-09-16 13:43:59.02434744 +0000 UTC m=+72.987706271" Sep 16 13:43:59.100793 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.100749 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" podStartSLOduration=47.379877081 podStartE2EDuration="55.100733611s" totalImagesPullingTime="7.72085653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:59.099637927 +0000 UTC m=+73.062996765" watchObservedRunningTime="2026-09-16 13:43:59.100733611 +0000 UTC m=+73.064092437" Sep 16 13:43:59.173812 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.173747 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e939fe78-e2d3-4469-9b2f-4a9f1b769ccf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hxdlt\" (UID: \"e939fe78-e2d3-4469-9b2f-4a9f1b769ccf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" Sep 16 13:43:59.275201 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.275169 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e939fe78-e2d3-4469-9b2f-4a9f1b769ccf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hxdlt\" (UID: \"e939fe78-e2d3-4469-9b2f-4a9f1b769ccf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" Sep 16 13:43:59.275343 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:59.275248 2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 16 13:43:59.275343 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:43:59.275320 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e939fe78-e2d3-4469-9b2f-4a9f1b769ccf-tls-certificates podName:e939fe78-e2d3-4469-9b2f-4a9f1b769ccf nodeName:}" failed. No retries permitted until 2026-09-16 13:43:59.775298784 +0000 UTC m=+73.738657594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e939fe78-e2d3-4469-9b2f-4a9f1b769ccf-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-hxdlt" (UID: "e939fe78-e2d3-4469-9b2f-4a9f1b769ccf") : secret "prometheus-operator-admission-webhook-tls" not found Sep 16 13:43:59.328866 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.328841 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-qsvk6" Sep 16 13:43:59.779459 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.779414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e939fe78-e2d3-4469-9b2f-4a9f1b769ccf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hxdlt\" (UID: \"e939fe78-e2d3-4469-9b2f-4a9f1b769ccf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" Sep 16 13:43:59.782892 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.782863 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e939fe78-e2d3-4469-9b2f-4a9f1b769ccf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hxdlt\" (UID: \"e939fe78-e2d3-4469-9b2f-4a9f1b769ccf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" Sep 16 13:43:59.925626 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:43:59.925596 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" Sep 16 13:44:00.227687 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:00.227656 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt"] Sep 16 13:44:00.230422 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:44:00.230391 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode939fe78_e2d3_4469_9b2f_4a9f1b769ccf.slice/crio-c1391008f12fbdad277a554a7c6fe641b0aa49a5d9c9ecbb9653f82f70164356 WatchSource:0}: Error finding container c1391008f12fbdad277a554a7c6fe641b0aa49a5d9c9ecbb9653f82f70164356: Status 404 returned error can't find the container with id c1391008f12fbdad277a554a7c6fe641b0aa49a5d9c9ecbb9653f82f70164356 Sep 16 13:44:00.913607 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:00.913570 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" event={"ID":"e939fe78-e2d3-4469-9b2f-4a9f1b769ccf","Type":"ContainerStarted","Data":"c1391008f12fbdad277a554a7c6fe641b0aa49a5d9c9ecbb9653f82f70164356"} Sep 16 13:44:01.919479 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:01.919393 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" event={"ID":"e939fe78-e2d3-4469-9b2f-4a9f1b769ccf","Type":"ContainerStarted","Data":"962272933f14d3ab20720dba853d38b4ede257829ab031388b5278cf34975d9e"} Sep 16 13:44:01.919888 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:01.919638 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" Sep 16 13:44:01.924298 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:01.924276 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" Sep 16 13:44:01.946828 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:01.946775 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hxdlt" podStartSLOduration=2.57216072 podStartE2EDuration="3.946757675s" totalImagesPullingTime="1.374596955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:01.946162875 +0000 UTC m=+75.909521695" watchObservedRunningTime="2026-09-16 13:44:01.946757675 +0000 UTC m=+75.910116501" Sep 16 13:44:08.132626 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.132596 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-989f686c8-r88cb"] Sep 16 13:44:08.135871 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.135849 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.138566 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.138544 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-59fzt\"" Sep 16 13:44:08.138566 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.138553 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 13:44:08.138704 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.138545 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 13:44:08.139619 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.139599 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 13:44:08.139720 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.139640 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 13:44:08.139720 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.139653 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 13:44:08.139720 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.139677 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 13:44:08.140262 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.140247 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 13:44:08.153658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.153637 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-989f686c8-r88cb"] Sep 16 13:44:08.245929 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.245899 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-service-ca\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.245929 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.245932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-oauth-serving-cert\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.246100 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.245991 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshgz\" (UniqueName: \"kubernetes.io/projected/b27a2f2d-160e-4534-bb4e-77eff0bf239e-kube-api-access-cshgz\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.246100 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.246029 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-config\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.246100 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.246057 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-oauth-config\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.246100 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.246081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-serving-cert\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.346921 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.346887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-oauth-config\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.346921 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.346920 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-serving-cert\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.347119 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.346957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-service-ca\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.347195 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.347137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-oauth-serving-cert\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.347271 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.347251 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshgz\" (UniqueName: \"kubernetes.io/projected/b27a2f2d-160e-4534-bb4e-77eff0bf239e-kube-api-access-cshgz\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.347325 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.347293 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-config\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.347746 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.347724 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-service-ca\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.347862 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.347852 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-oauth-serving-cert\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.348031 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.348010 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-config\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.349738 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.349713 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-oauth-config\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.349828 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.349714 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-serving-cert\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.366218 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.366199 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshgz\" (UniqueName: \"kubernetes.io/projected/b27a2f2d-160e-4534-bb4e-77eff0bf239e-kube-api-access-cshgz\") pod \"console-989f686c8-r88cb\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.445355 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.445299 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:08.750739 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.750636 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-989f686c8-r88cb"] Sep 16 13:44:08.753292 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:44:08.753267 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27a2f2d_160e_4534_bb4e_77eff0bf239e.slice/crio-0f55668961abf61cb92aff160a1edf6c8644c8354682d6e054a4ebf0311a4f1b WatchSource:0}: Error finding container 0f55668961abf61cb92aff160a1edf6c8644c8354682d6e054a4ebf0311a4f1b: Status 404 returned error can't find the container with id 0f55668961abf61cb92aff160a1edf6c8644c8354682d6e054a4ebf0311a4f1b Sep 16 13:44:08.909381 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.909345 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t2dsl" Sep 16 13:44:08.922688 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.922658 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q6bnh"] Sep 16 13:44:08.926958 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.926940 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:08.930278 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.930259 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 13:44:08.932434 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.932412 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5m2g2\"" Sep 16 13:44:08.933152 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.933120 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 13:44:08.933253 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.933229 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 13:44:08.933253 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.933234 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 13:44:08.946839 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:08.946815 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-989f686c8-r88cb" event={"ID":"b27a2f2d-160e-4534-bb4e-77eff0bf239e","Type":"ContainerStarted","Data":"0f55668961abf61cb92aff160a1edf6c8644c8354682d6e054a4ebf0311a4f1b"} Sep 16 13:44:09.054123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.053970 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-wtmp\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.054123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.054033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2faeae41-30b0-474c-b2b9-89109774172e-sys\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.054123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.054076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2faeae41-30b0-474c-b2b9-89109774172e-metrics-client-ca\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.054357 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.054162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-accelerators-collector-config\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.054357 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.054187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.054357 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.054235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2faeae41-30b0-474c-b2b9-89109774172e-root\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.054357 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.054257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52jw4\" (UniqueName: \"kubernetes.io/projected/2faeae41-30b0-474c-b2b9-89109774172e-kube-api-access-52jw4\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.054357 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.054350 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-textfile\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.054599 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.054372 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-tls\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155296 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-textfile\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155296 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155296 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-tls\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-wtmp\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155338 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2faeae41-30b0-474c-b2b9-89109774172e-sys\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2faeae41-30b0-474c-b2b9-89109774172e-metrics-client-ca\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155377 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-accelerators-collector-config\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155402 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155434 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2faeae41-30b0-474c-b2b9-89109774172e-sys\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155480 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2faeae41-30b0-474c-b2b9-89109774172e-root\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155481 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-wtmp\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2faeae41-30b0-474c-b2b9-89109774172e-root\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.155790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52jw4\" (UniqueName: \"kubernetes.io/projected/2faeae41-30b0-474c-b2b9-89109774172e-kube-api-access-52jw4\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.156180 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.155842 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-textfile\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.156229 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.156174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2faeae41-30b0-474c-b2b9-89109774172e-metrics-client-ca\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.156229 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.156187 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-accelerators-collector-config\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.157691 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.157674 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-tls\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.157812 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.157795 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2faeae41-30b0-474c-b2b9-89109774172e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.187785 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.187753 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52jw4\" (UniqueName: \"kubernetes.io/projected/2faeae41-30b0-474c-b2b9-89109774172e-kube-api-access-52jw4\") pod \"node-exporter-q6bnh\" (UID: \"2faeae41-30b0-474c-b2b9-89109774172e\") " pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.237353 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.237326 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q6bnh" Sep 16 13:44:09.248010 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:44:09.247986 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2faeae41_30b0_474c_b2b9_89109774172e.slice/crio-a124957b97e255b6f6d509199fd537c9cd84806e8b53164533aa012227e320cb WatchSource:0}: Error finding container a124957b97e255b6f6d509199fd537c9cd84806e8b53164533aa012227e320cb: Status 404 returned error can't find the container with id a124957b97e255b6f6d509199fd537c9cd84806e8b53164533aa012227e320cb Sep 16 13:44:09.953416 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:09.953386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6bnh" event={"ID":"2faeae41-30b0-474c-b2b9-89109774172e","Type":"ContainerStarted","Data":"a124957b97e255b6f6d509199fd537c9cd84806e8b53164533aa012227e320cb"} Sep 16 13:44:10.960017 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:10.959980 2581 generic.go:356] "Generic (PLEG): container finished" podID="2faeae41-30b0-474c-b2b9-89109774172e" containerID="97c9414cc959c2ed54e9b55fae14fae287049a46d24a18c35fd043da8c7a9b59" exitCode=0 Sep 16 13:44:10.960517 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:10.960050 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6bnh" event={"ID":"2faeae41-30b0-474c-b2b9-89109774172e","Type":"ContainerDied","Data":"97c9414cc959c2ed54e9b55fae14fae287049a46d24a18c35fd043da8c7a9b59"} Sep 16 13:44:11.969947 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:11.969848 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-989f686c8-r88cb" event={"ID":"b27a2f2d-160e-4534-bb4e-77eff0bf239e","Type":"ContainerStarted","Data":"3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362"} Sep 16 13:44:11.972083 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:11.972052 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6bnh" event={"ID":"2faeae41-30b0-474c-b2b9-89109774172e","Type":"ContainerStarted","Data":"03aa5ff0ee9e17e39c4d1e31596d014e2647bef2b4e83efefb3385e2cc5fa3b1"} Sep 16 13:44:11.972250 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:11.972092 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6bnh" event={"ID":"2faeae41-30b0-474c-b2b9-89109774172e","Type":"ContainerStarted","Data":"89d89565da8925d5a59bb3321b05bbefa38f33243d60d3afffa0e0d323fcc87f"} Sep 16 13:44:12.009964 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:12.009909 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-989f686c8-r88cb" podStartSLOduration=1.001177406 podStartE2EDuration="4.009891265s" totalImagesPullingTime="3.008713859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:12.00875605 +0000 UTC m=+85.972114879" watchObservedRunningTime="2026-09-16 13:44:12.009891265 +0000 UTC m=+85.973250095" Sep 16 13:44:12.044913 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:12.044866 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q6bnh" podStartSLOduration=3.042104894 podStartE2EDuration="4.044852276s" totalImagesPullingTime="932.8296ms" totalInitContainerRuntime="69.917782ms" isStatefulPod=true podCreationTimestamp="2026-09-16 13:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:12.043494125 +0000 UTC m=+86.006852951" watchObservedRunningTime="2026-09-16 13:44:12.044852276 +0000 UTC m=+86.008211102" Sep 16 13:44:13.342811 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.342782 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw"] Sep 16 13:44:13.346996 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.346979 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" Sep 16 13:44:13.349754 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.349728 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 16 13:44:13.349916 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.349900 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2tzsc\"" Sep 16 13:44:13.368403 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.368377 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw"] Sep 16 13:44:13.389012 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.388990 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49p7t\" (UniqueName: \"kubernetes.io/projected/9036dbc1-fab1-4f82-884d-5f012481128a-kube-api-access-49p7t\") pod \"monitoring-plugin-5d85778b9d-bb8mw\" (UID: \"9036dbc1-fab1-4f82-884d-5f012481128a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" Sep 16 13:44:13.389100 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.389052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9036dbc1-fab1-4f82-884d-5f012481128a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bb8mw\" (UID: \"9036dbc1-fab1-4f82-884d-5f012481128a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" Sep 16 13:44:13.489678 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.489656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9036dbc1-fab1-4f82-884d-5f012481128a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bb8mw\" (UID: \"9036dbc1-fab1-4f82-884d-5f012481128a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" Sep 16 13:44:13.489771 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.489689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49p7t\" (UniqueName: \"kubernetes.io/projected/9036dbc1-fab1-4f82-884d-5f012481128a-kube-api-access-49p7t\") pod \"monitoring-plugin-5d85778b9d-bb8mw\" (UID: \"9036dbc1-fab1-4f82-884d-5f012481128a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" Sep 16 13:44:13.492040 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.492019 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9036dbc1-fab1-4f82-884d-5f012481128a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bb8mw\" (UID: \"9036dbc1-fab1-4f82-884d-5f012481128a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" Sep 16 13:44:13.504495 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.504469 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49p7t\" (UniqueName: \"kubernetes.io/projected/9036dbc1-fab1-4f82-884d-5f012481128a-kube-api-access-49p7t\") pod \"monitoring-plugin-5d85778b9d-bb8mw\" (UID: \"9036dbc1-fab1-4f82-884d-5f012481128a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" Sep 16 13:44:13.655375 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.655302 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" Sep 16 13:44:13.761266 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.761235 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76f655d865-c75lb"] Sep 16 13:44:13.764518 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.764502 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.768758 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.768737 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 13:44:13.769238 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.769218 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 16 13:44:13.769312 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.769220 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-r49hf\"" Sep 16 13:44:13.770347 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.770328 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 16 13:44:13.770410 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.770366 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-sgg8qq4d3iq5\"" Sep 16 13:44:13.770459 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.770407 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 16 13:44:13.777537 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.777517 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76f655d865-c75lb"] Sep 16 13:44:13.792092 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.792058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8686d26c-77d7-4f8e-93dd-ca5d330dd679-secret-metrics-server-client-certs\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.792217 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.792127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljn9f\" (UniqueName: \"kubernetes.io/projected/8686d26c-77d7-4f8e-93dd-ca5d330dd679-kube-api-access-ljn9f\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.792217 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.792181 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8686d26c-77d7-4f8e-93dd-ca5d330dd679-secret-metrics-server-tls\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.792323 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.792219 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8686d26c-77d7-4f8e-93dd-ca5d330dd679-audit-log\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.792323 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.792271 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8686d26c-77d7-4f8e-93dd-ca5d330dd679-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.792323 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.792316 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8686d26c-77d7-4f8e-93dd-ca5d330dd679-metrics-server-audit-profiles\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.792468 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.792387 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8686d26c-77d7-4f8e-93dd-ca5d330dd679-client-ca-bundle\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.892942 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.892903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8686d26c-77d7-4f8e-93dd-ca5d330dd679-metrics-server-audit-profiles\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.892942 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.892950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8686d26c-77d7-4f8e-93dd-ca5d330dd679-client-ca-bundle\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.893132 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.892980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8686d26c-77d7-4f8e-93dd-ca5d330dd679-secret-metrics-server-client-certs\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.893132 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.893006 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljn9f\" (UniqueName: \"kubernetes.io/projected/8686d26c-77d7-4f8e-93dd-ca5d330dd679-kube-api-access-ljn9f\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.893132 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.893028 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8686d26c-77d7-4f8e-93dd-ca5d330dd679-secret-metrics-server-tls\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.893132 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.893050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8686d26c-77d7-4f8e-93dd-ca5d330dd679-audit-log\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.893132 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.893075 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8686d26c-77d7-4f8e-93dd-ca5d330dd679-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.893590 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.893565 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8686d26c-77d7-4f8e-93dd-ca5d330dd679-audit-log\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.893842 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.893818 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8686d26c-77d7-4f8e-93dd-ca5d330dd679-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.894030 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.894008 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8686d26c-77d7-4f8e-93dd-ca5d330dd679-metrics-server-audit-profiles\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.895511 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.895486 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8686d26c-77d7-4f8e-93dd-ca5d330dd679-client-ca-bundle\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.895628 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.895556 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8686d26c-77d7-4f8e-93dd-ca5d330dd679-secret-metrics-server-tls\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.895686 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.895668 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8686d26c-77d7-4f8e-93dd-ca5d330dd679-secret-metrics-server-client-certs\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.907993 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.907942 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljn9f\" (UniqueName: \"kubernetes.io/projected/8686d26c-77d7-4f8e-93dd-ca5d330dd679-kube-api-access-ljn9f\") pod \"metrics-server-76f655d865-c75lb\" (UID: \"8686d26c-77d7-4f8e-93dd-ca5d330dd679\") " pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:13.930303 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.930281 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw"] Sep 16 13:44:13.932611 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:44:13.932586 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9036dbc1_fab1_4f82_884d_5f012481128a.slice/crio-494a2ff69dce49a820c158f3e4dd8c37c7861bb114e1ecf6dfd272bc3cbd4617 WatchSource:0}: Error finding container 494a2ff69dce49a820c158f3e4dd8c37c7861bb114e1ecf6dfd272bc3cbd4617: Status 404 returned error can't find the container with id 494a2ff69dce49a820c158f3e4dd8c37c7861bb114e1ecf6dfd272bc3cbd4617 Sep 16 13:44:13.980980 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:13.980943 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" event={"ID":"9036dbc1-fab1-4f82-884d-5f012481128a","Type":"ContainerStarted","Data":"494a2ff69dce49a820c158f3e4dd8c37c7861bb114e1ecf6dfd272bc3cbd4617"} Sep 16 13:44:14.073685 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.073651 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:14.126319 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.126287 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55fc589b5d-8m9rl"] Sep 16 13:44:14.130631 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.130607 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.133401 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.133372 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 16 13:44:14.133956 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.133644 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 16 13:44:14.133956 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.133653 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-j64zd\"" Sep 16 13:44:14.133956 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.133802 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 16 13:44:14.133956 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.133846 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 16 13:44:14.133956 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.133910 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 16 13:44:14.139028 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.139003 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 16 13:44:14.144643 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.144621 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55fc589b5d-8m9rl"] Sep 16 13:44:14.194709 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.194646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-telemeter-client-tls\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.194709 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.194680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.194847 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.194717 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-secret-telemeter-client\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.194847 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.194771 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-federate-client-tls\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.194847 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.194808 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmc2\" (UniqueName: \"kubernetes.io/projected/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-kube-api-access-slmc2\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.194847 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.194832 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-serving-certs-ca-bundle\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.194971 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.194873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-metrics-client-ca\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.194971 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.194927 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.296178 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.296125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-telemeter-client-tls\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.296328 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.296187 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.296328 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.296222 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-secret-telemeter-client\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.296328 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.296250 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-federate-client-tls\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.296328 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.296278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmc2\" (UniqueName: \"kubernetes.io/projected/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-kube-api-access-slmc2\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.296532 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.296342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-serving-certs-ca-bundle\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.296532 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.296421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-metrics-client-ca\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.296532 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.296479 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.297257 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.297201 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-metrics-client-ca\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.297257 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.297224 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-serving-certs-ca-bundle\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.297454 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.297433 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.298742 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.298717 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-telemeter-client-tls\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.298843 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.298788 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-secret-telemeter-client\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.299042 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.299023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.299207 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.299188 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-federate-client-tls\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.309040 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.309013 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmc2\" (UniqueName: \"kubernetes.io/projected/42f70be2-7a7a-4cd8-b95d-bfec9420a78d-kube-api-access-slmc2\") pod \"telemeter-client-55fc589b5d-8m9rl\" (UID: \"42f70be2-7a7a-4cd8-b95d-bfec9420a78d\") " pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.363330 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.363226 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76f655d865-c75lb"] Sep 16 13:44:14.367220 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:44:14.367191 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8686d26c_77d7_4f8e_93dd_ca5d330dd679.slice/crio-ed7cd2d5a5446529e5a36504fa5fa700fc5c47f3ed5732c019935baaeb3ff600 WatchSource:0}: Error finding container ed7cd2d5a5446529e5a36504fa5fa700fc5c47f3ed5732c019935baaeb3ff600: Status 404 returned error can't find the container with id ed7cd2d5a5446529e5a36504fa5fa700fc5c47f3ed5732c019935baaeb3ff600 Sep 16 13:44:14.440779 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.440749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" Sep 16 13:44:14.738421 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.738392 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55fc589b5d-8m9rl"] Sep 16 13:44:14.989294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:14.989200 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76f655d865-c75lb" event={"ID":"8686d26c-77d7-4f8e-93dd-ca5d330dd679","Type":"ContainerStarted","Data":"ed7cd2d5a5446529e5a36504fa5fa700fc5c47f3ed5732c019935baaeb3ff600"} Sep 16 13:44:15.130236 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:44:15.130192 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f70be2_7a7a_4cd8_b95d_bfec9420a78d.slice/crio-127e60b760e5f52eb75b5f8983adefeb1db455efd3ca033244a5b43554fec56e WatchSource:0}: Error finding container 127e60b760e5f52eb75b5f8983adefeb1db455efd3ca033244a5b43554fec56e: Status 404 returned error can't find the container with id 127e60b760e5f52eb75b5f8983adefeb1db455efd3ca033244a5b43554fec56e Sep 16 13:44:15.874998 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.874961 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8447d77ccb-2jv59"] Sep 16 13:44:15.878398 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.878337 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:15.886820 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.886797 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 13:44:15.894110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.894087 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8447d77ccb-2jv59"] Sep 16 13:44:15.911103 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.911078 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r44cj\" (UniqueName: \"kubernetes.io/projected/975ee0d1-2961-44a3-83ea-a6f511831404-kube-api-access-r44cj\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:15.911236 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.911111 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-console-config\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:15.911236 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.911156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-service-ca\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:15.911236 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.911211 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-oauth-serving-cert\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:15.911397 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.911249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-serving-cert\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:15.911397 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.911279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-oauth-config\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:15.911397 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.911296 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-trusted-ca-bundle\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:15.996010 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.995970 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" event={"ID":"9036dbc1-fab1-4f82-884d-5f012481128a","Type":"ContainerStarted","Data":"93e2442922f73c0fdf725ee4811b52570aeb872c44cce5e8659d784eb3ac8f68"} Sep 16 13:44:15.996185 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.996044 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" Sep 16 13:44:15.997769 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:15.997741 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" event={"ID":"42f70be2-7a7a-4cd8-b95d-bfec9420a78d","Type":"ContainerStarted","Data":"127e60b760e5f52eb75b5f8983adefeb1db455efd3ca033244a5b43554fec56e"} Sep 16 13:44:16.000626 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.000600 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76f655d865-c75lb" event={"ID":"8686d26c-77d7-4f8e-93dd-ca5d330dd679","Type":"ContainerStarted","Data":"2b96b70606f54024d978a1787091be4265acd402d8e2faf9fd6a8326f4008b15"} Sep 16 13:44:16.002056 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.002028 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" Sep 16 13:44:16.012624 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.012591 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r44cj\" (UniqueName: \"kubernetes.io/projected/975ee0d1-2961-44a3-83ea-a6f511831404-kube-api-access-r44cj\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.012725 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.012638 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-console-config\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.012725 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.012668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-service-ca\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.012831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.012736 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-oauth-serving-cert\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.012831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.012775 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-serving-cert\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.012831 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.012816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-oauth-config\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.012935 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.012841 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-trusted-ca-bundle\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.013817 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.013741 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-trusted-ca-bundle\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.014635 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.014550 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-oauth-serving-cert\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.014768 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.014749 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-console-config\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.015762 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.015736 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-service-ca\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.017407 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.017384 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-serving-cert\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.017769 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.017749 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-oauth-config\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.026243 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.026221 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r44cj\" (UniqueName: \"kubernetes.io/projected/975ee0d1-2961-44a3-83ea-a6f511831404-kube-api-access-r44cj\") pod \"console-8447d77ccb-2jv59\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.030693 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.030648 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bb8mw" podStartSLOduration=1.779863093 podStartE2EDuration="3.030632067s" totalImagesPullingTime="1.250768974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:16.030055303 +0000 UTC m=+89.993414146" watchObservedRunningTime="2026-09-16 13:44:16.030632067 +0000 UTC m=+89.993990894" Sep 16 13:44:16.072900 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.072852 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76f655d865-c75lb" podStartSLOduration=1.813840671 podStartE2EDuration="3.072834388s" totalImagesPullingTime="1.258993717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:16.07266834 +0000 UTC m=+90.036027166" watchObservedRunningTime="2026-09-16 13:44:16.072834388 +0000 UTC m=+90.036193216" Sep 16 13:44:16.189832 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.189750 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:16.508677 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:16.508642 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8447d77ccb-2jv59"] Sep 16 13:44:16.844596 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:44:16.844521 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975ee0d1_2961_44a3_83ea_a6f511831404.slice/crio-1dd54e616d41d1d1257df18b8e27a43cd925cc5852a681c747c95397a0888eca WatchSource:0}: Error finding container 1dd54e616d41d1d1257df18b8e27a43cd925cc5852a681c747c95397a0888eca: Status 404 returned error can't find the container with id 1dd54e616d41d1d1257df18b8e27a43cd925cc5852a681c747c95397a0888eca Sep 16 13:44:17.006162 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:17.006111 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" event={"ID":"42f70be2-7a7a-4cd8-b95d-bfec9420a78d","Type":"ContainerStarted","Data":"e2461fd0652828eb9d48fa61b042b2e77a436ec4fb3757ad93cabf92aec355bc"} Sep 16 13:44:17.007881 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:17.007806 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8447d77ccb-2jv59" event={"ID":"975ee0d1-2961-44a3-83ea-a6f511831404","Type":"ContainerStarted","Data":"66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c"} Sep 16 13:44:17.007881 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:17.007854 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8447d77ccb-2jv59" event={"ID":"975ee0d1-2961-44a3-83ea-a6f511831404","Type":"ContainerStarted","Data":"1dd54e616d41d1d1257df18b8e27a43cd925cc5852a681c747c95397a0888eca"} Sep 16 13:44:17.022625 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:17.022604 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:44:17.045433 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:17.045386 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8447d77ccb-2jv59" podStartSLOduration=2.045373264 podStartE2EDuration="2.045373264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:17.043320473 +0000 UTC m=+91.006679300" watchObservedRunningTime="2026-09-16 13:44:17.045373264 +0000 UTC m=+91.008732090" Sep 16 13:44:18.013931 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:18.013899 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" event={"ID":"42f70be2-7a7a-4cd8-b95d-bfec9420a78d","Type":"ContainerStarted","Data":"72065f238f1e1388f33f8a9ec021b3c09fe5e87454778050e4e41bdcc429b19e"} Sep 16 13:44:18.014389 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:18.013956 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" event={"ID":"42f70be2-7a7a-4cd8-b95d-bfec9420a78d","Type":"ContainerStarted","Data":"ac8b6297278343fef1053708aa4893c6a05403e51ea25c513849e9912089c990"} Sep 16 13:44:18.039292 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:18.039240 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55fc589b5d-8m9rl" podStartSLOduration=1.409695245 podStartE2EDuration="4.039222267s" totalImagesPullingTime="2.629527022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:18.037514068 +0000 UTC m=+92.000872923" watchObservedRunningTime="2026-09-16 13:44:18.039222267 +0000 UTC m=+92.002581095" Sep 16 13:44:18.445806 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:18.445763 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:18.445996 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:18.445838 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:18.450648 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:18.450626 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:19.021967 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:19.021940 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:21.480634 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.480594 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8447d77ccb-2jv59"] Sep 16 13:44:21.524935 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.524903 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8d8566bd5-56jpv"] Sep 16 13:44:21.530493 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.530469 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.542711 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.542691 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d8566bd5-56jpv"] Sep 16 13:44:21.558878 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.558859 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-oauth-config\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.558963 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.558885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-service-ca\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.558963 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.558908 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rpk\" (UniqueName: \"kubernetes.io/projected/002b49a4-3115-46ca-a3fb-4f3062965de3-kube-api-access-m7rpk\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.558963 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.558932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-oauth-serving-cert\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.559055 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.559004 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-trusted-ca-bundle\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.559118 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.559099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-console-config\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.559172 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.559158 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-serving-cert\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.659995 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.659972 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-console-config\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.660078 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.660004 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-serving-cert\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.660078 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.660026 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-oauth-config\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.660078 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.660043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-service-ca\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.660078 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.660067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rpk\" (UniqueName: \"kubernetes.io/projected/002b49a4-3115-46ca-a3fb-4f3062965de3-kube-api-access-m7rpk\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.660294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.660104 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-oauth-serving-cert\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.660294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.660134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-trusted-ca-bundle\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.660758 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.660735 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-console-config\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.660868 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.660845 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-service-ca\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.660930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.660888 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-oauth-serving-cert\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.660986 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.660966 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-trusted-ca-bundle\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.662510 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.662479 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-serving-cert\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.662587 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.662568 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-oauth-config\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.673303 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.673283 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rpk\" (UniqueName: \"kubernetes.io/projected/002b49a4-3115-46ca-a3fb-4f3062965de3-kube-api-access-m7rpk\") pod \"console-8d8566bd5-56jpv\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:21.838827 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:21.838756 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:22.037049 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.037015 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" podUID="28371faa-0a08-49cf-8a63-9abe68aca3bc" containerName="registry" containerID="cri-o://5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0" gracePeriod=30 Sep 16 13:44:22.113784 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.113762 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d8566bd5-56jpv"] Sep 16 13:44:22.116245 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:44:22.116219 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002b49a4_3115_46ca_a3fb_4f3062965de3.slice/crio-74064a6d5cf710179ed95fac544cb9bd386a4eafb0b1919e960ed052662cde1e WatchSource:0}: Error finding container 74064a6d5cf710179ed95fac544cb9bd386a4eafb0b1919e960ed052662cde1e: Status 404 returned error can't find the container with id 74064a6d5cf710179ed95fac544cb9bd386a4eafb0b1919e960ed052662cde1e Sep 16 13:44:22.271233 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.271201 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:44:22.364083 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.364014 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-trusted-ca\") pod \"28371faa-0a08-49cf-8a63-9abe68aca3bc\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " Sep 16 13:44:22.364083 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.364056 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-image-registry-private-configuration\") pod \"28371faa-0a08-49cf-8a63-9abe68aca3bc\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " Sep 16 13:44:22.364294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.364092 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cph2f\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-kube-api-access-cph2f\") pod \"28371faa-0a08-49cf-8a63-9abe68aca3bc\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " Sep 16 13:44:22.364294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.364125 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-certificates\") pod \"28371faa-0a08-49cf-8a63-9abe68aca3bc\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " Sep 16 13:44:22.364294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.364182 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28371faa-0a08-49cf-8a63-9abe68aca3bc-ca-trust-extracted\") pod \"28371faa-0a08-49cf-8a63-9abe68aca3bc\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " Sep 16 13:44:22.364294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.364215 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") pod \"28371faa-0a08-49cf-8a63-9abe68aca3bc\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " Sep 16 13:44:22.364294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.364249 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-bound-sa-token\") pod \"28371faa-0a08-49cf-8a63-9abe68aca3bc\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " Sep 16 13:44:22.364294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.364292 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-installation-pull-secrets\") pod \"28371faa-0a08-49cf-8a63-9abe68aca3bc\" (UID: \"28371faa-0a08-49cf-8a63-9abe68aca3bc\") " Sep 16 13:44:22.364606 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.364577 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "28371faa-0a08-49cf-8a63-9abe68aca3bc" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:22.364734 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.364601 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "28371faa-0a08-49cf-8a63-9abe68aca3bc" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:22.366688 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.366656 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-kube-api-access-cph2f" (OuterVolumeSpecName: "kube-api-access-cph2f") pod "28371faa-0a08-49cf-8a63-9abe68aca3bc" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc"). InnerVolumeSpecName "kube-api-access-cph2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:22.366832 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.366763 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "28371faa-0a08-49cf-8a63-9abe68aca3bc" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:44:22.366832 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.366801 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "28371faa-0a08-49cf-8a63-9abe68aca3bc" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:22.367003 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.366983 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "28371faa-0a08-49cf-8a63-9abe68aca3bc" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:22.367059 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.366999 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "28371faa-0a08-49cf-8a63-9abe68aca3bc" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:44:22.372100 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.372079 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28371faa-0a08-49cf-8a63-9abe68aca3bc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "28371faa-0a08-49cf-8a63-9abe68aca3bc" (UID: "28371faa-0a08-49cf-8a63-9abe68aca3bc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:44:22.465688 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.465660 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-trusted-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:22.465688 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.465683 2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-image-registry-private-configuration\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:22.465688 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.465693 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cph2f\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-kube-api-access-cph2f\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:22.465877 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.465702 2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-certificates\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:22.465877 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.465710 2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28371faa-0a08-49cf-8a63-9abe68aca3bc-ca-trust-extracted\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:22.465877 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.465718 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-registry-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:22.465877 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.465726 2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28371faa-0a08-49cf-8a63-9abe68aca3bc-bound-sa-token\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:22.465877 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:22.465734 2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28371faa-0a08-49cf-8a63-9abe68aca3bc-installation-pull-secrets\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:23.036464 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.036423 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d8566bd5-56jpv" event={"ID":"002b49a4-3115-46ca-a3fb-4f3062965de3","Type":"ContainerStarted","Data":"20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d"} Sep 16 13:44:23.036860 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.036478 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d8566bd5-56jpv" event={"ID":"002b49a4-3115-46ca-a3fb-4f3062965de3","Type":"ContainerStarted","Data":"74064a6d5cf710179ed95fac544cb9bd386a4eafb0b1919e960ed052662cde1e"} Sep 16 13:44:23.037741 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.037724 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" Sep 16 13:44:23.037805 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.037740 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" event={"ID":"28371faa-0a08-49cf-8a63-9abe68aca3bc","Type":"ContainerDied","Data":"5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0"} Sep 16 13:44:23.037805 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.037781 2581 scope.go:117] "RemoveContainer" containerID="5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0" Sep 16 13:44:23.037885 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.037716 2581 generic.go:356] "Generic (PLEG): container finished" podID="28371faa-0a08-49cf-8a63-9abe68aca3bc" containerID="5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0" exitCode=0 Sep 16 13:44:23.037885 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.037852 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67fcbbf884-59lgn" event={"ID":"28371faa-0a08-49cf-8a63-9abe68aca3bc","Type":"ContainerDied","Data":"863e9bd8b23adec72c914b1461a79c24f8639a5c032350dff1663bc1ad8a6725"} Sep 16 13:44:23.049264 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.049242 2581 scope.go:117] "RemoveContainer" containerID="5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0" Sep 16 13:44:23.049587 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:44:23.049567 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0\": container with ID starting with 5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0 not found: ID does not exist" containerID="5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0" Sep 16 13:44:23.049666 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.049594 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0"} err="failed to get container status \"5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0\": rpc error: code = NotFound desc = could not find container \"5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0\": container with ID starting with 5b8a49d34cd9b7b342c3d5d272cca8773ed9338a815dcb605f6a0262e95e6fc0 not found: ID does not exist" Sep 16 13:44:23.056350 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.056313 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8d8566bd5-56jpv" podStartSLOduration=2.056300826 podStartE2EDuration="2.056300826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:23.055449305 +0000 UTC m=+97.018808133" watchObservedRunningTime="2026-09-16 13:44:23.056300826 +0000 UTC m=+97.019659652" Sep 16 13:44:23.075562 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.075542 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67fcbbf884-59lgn"] Sep 16 13:44:23.085519 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:23.085502 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67fcbbf884-59lgn"] Sep 16 13:44:24.516257 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:24.516226 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28371faa-0a08-49cf-8a63-9abe68aca3bc" path="/var/lib/kubelet/pods/28371faa-0a08-49cf-8a63-9abe68aca3bc/volumes" Sep 16 13:44:25.880828 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:25.880799 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mcl5r" Sep 16 13:44:26.190019 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:26.189955 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:31.839799 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:31.839765 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:31.840205 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:31.839814 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:31.844696 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:31.844665 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:32.080655 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:32.080627 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:44:32.136498 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:32.136414 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-989f686c8-r88cb"] Sep 16 13:44:34.074511 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:34.074478 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:34.074970 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:34.074544 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:42.120540 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:42.120502 2581 generic.go:356] "Generic (PLEG): container finished" podID="9590e744-474c-4da6-a84f-eee73056b9c9" containerID="26ea916afc9c2c59d27ce2824ebb50c964555df95cee8237cec65a28e2ff9f6f" exitCode=0 Sep 16 13:44:42.120540 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:42.120543 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" event={"ID":"9590e744-474c-4da6-a84f-eee73056b9c9","Type":"ContainerDied","Data":"26ea916afc9c2c59d27ce2824ebb50c964555df95cee8237cec65a28e2ff9f6f"} Sep 16 13:44:42.121054 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:42.120854 2581 scope.go:117] "RemoveContainer" containerID="26ea916afc9c2c59d27ce2824ebb50c964555df95cee8237cec65a28e2ff9f6f" Sep 16 13:44:43.126314 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:43.126278 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69jqf" event={"ID":"9590e744-474c-4da6-a84f-eee73056b9c9","Type":"ContainerStarted","Data":"08fa0159f025f98674616c9cd28c1eef4be8da537fe67f81ad87d07c07ab1a5d"} Sep 16 13:44:46.500209 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.500170 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8447d77ccb-2jv59" podUID="975ee0d1-2961-44a3-83ea-a6f511831404" containerName="console" containerID="cri-o://66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c" gracePeriod=15 Sep 16 13:44:46.779065 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.779041 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:46.859322 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859293 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r44cj\" (UniqueName: \"kubernetes.io/projected/975ee0d1-2961-44a3-83ea-a6f511831404-kube-api-access-r44cj\") pod \"975ee0d1-2961-44a3-83ea-a6f511831404\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " Sep 16 13:44:46.859469 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859330 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-serving-cert\") pod \"975ee0d1-2961-44a3-83ea-a6f511831404\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " Sep 16 13:44:46.859469 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859374 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-trusted-ca-bundle\") pod \"975ee0d1-2961-44a3-83ea-a6f511831404\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " Sep 16 13:44:46.859469 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859400 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-oauth-config\") pod \"975ee0d1-2961-44a3-83ea-a6f511831404\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " Sep 16 13:44:46.859469 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859449 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-service-ca\") pod \"975ee0d1-2961-44a3-83ea-a6f511831404\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " Sep 16 13:44:46.859663 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859473 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-oauth-serving-cert\") pod \"975ee0d1-2961-44a3-83ea-a6f511831404\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " Sep 16 13:44:46.859663 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859519 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-console-config\") pod \"975ee0d1-2961-44a3-83ea-a6f511831404\" (UID: \"975ee0d1-2961-44a3-83ea-a6f511831404\") " Sep 16 13:44:46.859923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859894 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-service-ca" (OuterVolumeSpecName: "service-ca") pod "975ee0d1-2961-44a3-83ea-a6f511831404" (UID: "975ee0d1-2961-44a3-83ea-a6f511831404"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:46.859923 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859909 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "975ee0d1-2961-44a3-83ea-a6f511831404" (UID: "975ee0d1-2961-44a3-83ea-a6f511831404"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:46.860065 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859900 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "975ee0d1-2961-44a3-83ea-a6f511831404" (UID: "975ee0d1-2961-44a3-83ea-a6f511831404"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:46.860065 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.859946 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-console-config" (OuterVolumeSpecName: "console-config") pod "975ee0d1-2961-44a3-83ea-a6f511831404" (UID: "975ee0d1-2961-44a3-83ea-a6f511831404"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:46.860130 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.860068 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-console-config\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:46.860130 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.860081 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-trusted-ca-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:46.860130 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.860091 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:46.860130 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.860100 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/975ee0d1-2961-44a3-83ea-a6f511831404-oauth-serving-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:46.861663 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.861637 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "975ee0d1-2961-44a3-83ea-a6f511831404" (UID: "975ee0d1-2961-44a3-83ea-a6f511831404"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:44:46.861781 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.861762 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975ee0d1-2961-44a3-83ea-a6f511831404-kube-api-access-r44cj" (OuterVolumeSpecName: "kube-api-access-r44cj") pod "975ee0d1-2961-44a3-83ea-a6f511831404" (UID: "975ee0d1-2961-44a3-83ea-a6f511831404"). InnerVolumeSpecName "kube-api-access-r44cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:46.861849 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.861765 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "975ee0d1-2961-44a3-83ea-a6f511831404" (UID: "975ee0d1-2961-44a3-83ea-a6f511831404"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:44:46.960539 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.960506 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r44cj\" (UniqueName: \"kubernetes.io/projected/975ee0d1-2961-44a3-83ea-a6f511831404-kube-api-access-r44cj\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:46.960539 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.960537 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-serving-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:46.960691 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:46.960550 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/975ee0d1-2961-44a3-83ea-a6f511831404-console-oauth-config\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:47.143420 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:47.143345 2581 generic.go:356] "Generic (PLEG): container finished" podID="975ee0d1-2961-44a3-83ea-a6f511831404" containerID="66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c" exitCode=0 Sep 16 13:44:47.143420 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:47.143391 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8447d77ccb-2jv59" Sep 16 13:44:47.143602 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:47.143393 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8447d77ccb-2jv59" event={"ID":"975ee0d1-2961-44a3-83ea-a6f511831404","Type":"ContainerDied","Data":"66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c"} Sep 16 13:44:47.143602 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:47.143522 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8447d77ccb-2jv59" event={"ID":"975ee0d1-2961-44a3-83ea-a6f511831404","Type":"ContainerDied","Data":"1dd54e616d41d1d1257df18b8e27a43cd925cc5852a681c747c95397a0888eca"} Sep 16 13:44:47.143602 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:47.143551 2581 scope.go:117] "RemoveContainer" containerID="66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c" Sep 16 13:44:47.154784 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:47.154767 2581 scope.go:117] "RemoveContainer" containerID="66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c" Sep 16 13:44:47.155073 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:44:47.155050 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c\": container with ID starting with 66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c not found: ID does not exist" containerID="66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c" Sep 16 13:44:47.155115 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:47.155081 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c"} err="failed to get container status \"66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c\": rpc error: code = NotFound desc = could not find container \"66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c\": container with ID starting with 66b2f9725abfae5a86a56524a0f8015d9edc41fee3c88e6f75c6f92887551c1c not found: ID does not exist" Sep 16 13:44:47.171990 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:47.171964 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8447d77ccb-2jv59"] Sep 16 13:44:47.175967 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:47.175947 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8447d77ccb-2jv59"] Sep 16 13:44:48.514470 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:48.514438 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="975ee0d1-2961-44a3-83ea-a6f511831404" path="/var/lib/kubelet/pods/975ee0d1-2961-44a3-83ea-a6f511831404/volumes" Sep 16 13:44:54.078713 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:54.078683 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:54.082326 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:54.082304 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76f655d865-c75lb" Sep 16 13:44:56.182283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:56.182257 2581 generic.go:356] "Generic (PLEG): container finished" podID="80c00f5f-761c-4d82-a657-52e437aee43d" containerID="f599d02909da616216c9d109e389c485370289c30b1e868a58744c2c9fc759c8" exitCode=0 Sep 16 13:44:56.182555 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:56.182326 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" event={"ID":"80c00f5f-761c-4d82-a657-52e437aee43d","Type":"ContainerDied","Data":"f599d02909da616216c9d109e389c485370289c30b1e868a58744c2c9fc759c8"} Sep 16 13:44:56.182647 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:56.182633 2581 scope.go:117] "RemoveContainer" containerID="f599d02909da616216c9d109e389c485370289c30b1e868a58744c2c9fc759c8" Sep 16 13:44:57.159854 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.159810 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-989f686c8-r88cb" podUID="b27a2f2d-160e-4534-bb4e-77eff0bf239e" containerName="console" containerID="cri-o://3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362" gracePeriod=15 Sep 16 13:44:57.187827 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.187800 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-bknfg" event={"ID":"80c00f5f-761c-4d82-a657-52e437aee43d","Type":"ContainerStarted","Data":"3cddfc3c8bc9366299919208d768fbbfaf1b1aa846a09eb13e54e83e1546bf58"} Sep 16 13:44:57.400657 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.400629 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:57.536013 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.535985 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cshgz\" (UniqueName: \"kubernetes.io/projected/b27a2f2d-160e-4534-bb4e-77eff0bf239e-kube-api-access-cshgz\") pod \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " Sep 16 13:44:57.536180 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.536037 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-serving-cert\") pod \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " Sep 16 13:44:57.536180 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.536064 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-oauth-serving-cert\") pod \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " Sep 16 13:44:57.536308 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.536189 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-service-ca\") pod \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " Sep 16 13:44:57.536308 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.536232 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-config\") pod \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " Sep 16 13:44:57.536308 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.536273 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-oauth-config\") pod \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\" (UID: \"b27a2f2d-160e-4534-bb4e-77eff0bf239e\") " Sep 16 13:44:57.536518 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.536411 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b27a2f2d-160e-4534-bb4e-77eff0bf239e" (UID: "b27a2f2d-160e-4534-bb4e-77eff0bf239e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:57.536638 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.536587 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-config" (OuterVolumeSpecName: "console-config") pod "b27a2f2d-160e-4534-bb4e-77eff0bf239e" (UID: "b27a2f2d-160e-4534-bb4e-77eff0bf239e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:57.536638 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.536593 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-service-ca" (OuterVolumeSpecName: "service-ca") pod "b27a2f2d-160e-4534-bb4e-77eff0bf239e" (UID: "b27a2f2d-160e-4534-bb4e-77eff0bf239e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:57.538294 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.538266 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b27a2f2d-160e-4534-bb4e-77eff0bf239e" (UID: "b27a2f2d-160e-4534-bb4e-77eff0bf239e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:44:57.538624 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.538601 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b27a2f2d-160e-4534-bb4e-77eff0bf239e" (UID: "b27a2f2d-160e-4534-bb4e-77eff0bf239e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:44:57.538624 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.538614 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27a2f2d-160e-4534-bb4e-77eff0bf239e-kube-api-access-cshgz" (OuterVolumeSpecName: "kube-api-access-cshgz") pod "b27a2f2d-160e-4534-bb4e-77eff0bf239e" (UID: "b27a2f2d-160e-4534-bb4e-77eff0bf239e"). InnerVolumeSpecName "kube-api-access-cshgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:57.637773 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.637746 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:57.637773 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.637773 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-config\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:57.637897 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.637785 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-oauth-config\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:57.637897 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.637794 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cshgz\" (UniqueName: \"kubernetes.io/projected/b27a2f2d-160e-4534-bb4e-77eff0bf239e-kube-api-access-cshgz\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:57.637897 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.637804 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b27a2f2d-160e-4534-bb4e-77eff0bf239e-console-serving-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:57.637897 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:57.637812 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b27a2f2d-160e-4534-bb4e-77eff0bf239e-oauth-serving-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:44:58.192791 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:58.192753 2581 generic.go:356] "Generic (PLEG): container finished" podID="b27a2f2d-160e-4534-bb4e-77eff0bf239e" containerID="3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362" exitCode=0 Sep 16 13:44:58.193241 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:58.192803 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-989f686c8-r88cb" Sep 16 13:44:58.193241 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:58.192823 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-989f686c8-r88cb" event={"ID":"b27a2f2d-160e-4534-bb4e-77eff0bf239e","Type":"ContainerDied","Data":"3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362"} Sep 16 13:44:58.193241 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:58.192872 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-989f686c8-r88cb" event={"ID":"b27a2f2d-160e-4534-bb4e-77eff0bf239e","Type":"ContainerDied","Data":"0f55668961abf61cb92aff160a1edf6c8644c8354682d6e054a4ebf0311a4f1b"} Sep 16 13:44:58.193241 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:58.192902 2581 scope.go:117] "RemoveContainer" containerID="3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362" Sep 16 13:44:58.203700 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:58.203624 2581 scope.go:117] "RemoveContainer" containerID="3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362" Sep 16 13:44:58.204295 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:44:58.204254 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362\": container with ID starting with 3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362 not found: ID does not exist" containerID="3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362" Sep 16 13:44:58.204401 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:58.204293 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362"} err="failed to get container status \"3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362\": rpc error: code = NotFound desc = could not find container \"3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362\": container with ID starting with 3d2e6375e2a9566ab4f4decf26d6463a5ef64fba7d5bc213df5604233fa7d362 not found: ID does not exist" Sep 16 13:44:58.217306 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:58.217286 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-989f686c8-r88cb"] Sep 16 13:44:58.222158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:58.222120 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-989f686c8-r88cb"] Sep 16 13:44:58.514002 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:44:58.513972 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b27a2f2d-160e-4534-bb4e-77eff0bf239e" path="/var/lib/kubelet/pods/b27a2f2d-160e-4534-bb4e-77eff0bf239e/volumes" Sep 16 13:45:07.229190 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:07.229133 2581 generic.go:356] "Generic (PLEG): container finished" podID="97f4c753-e774-452c-8c8d-4070badd940e" containerID="dcf9201a55b5f3e277baba70b09774ccf099058e3d3b78d5705f43737eb68447" exitCode=0 Sep 16 13:45:07.229592 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:07.229206 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6nvrv" event={"ID":"97f4c753-e774-452c-8c8d-4070badd940e","Type":"ContainerDied","Data":"dcf9201a55b5f3e277baba70b09774ccf099058e3d3b78d5705f43737eb68447"} Sep 16 13:45:07.229592 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:07.229532 2581 scope.go:117] "RemoveContainer" containerID="dcf9201a55b5f3e277baba70b09774ccf099058e3d3b78d5705f43737eb68447" Sep 16 13:45:08.235477 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:08.235441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6nvrv" event={"ID":"97f4c753-e774-452c-8c8d-4070badd940e","Type":"ContainerStarted","Data":"d8a9a8e9bad08a6c8abd8208905cb64e68fca950ed1471266b24493e2185d6df"} Sep 16 13:45:52.087130 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.087094 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f745c9f75-szlvf"] Sep 16 13:45:52.088136 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.087598 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b27a2f2d-160e-4534-bb4e-77eff0bf239e" containerName="console" Sep 16 13:45:52.088136 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.087626 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b27a2f2d-160e-4534-bb4e-77eff0bf239e" containerName="console" Sep 16 13:45:52.088136 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.087648 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28371faa-0a08-49cf-8a63-9abe68aca3bc" containerName="registry" Sep 16 13:45:52.088136 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.087658 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="28371faa-0a08-49cf-8a63-9abe68aca3bc" containerName="registry" Sep 16 13:45:52.088136 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.087670 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="975ee0d1-2961-44a3-83ea-a6f511831404" containerName="console" Sep 16 13:45:52.088136 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.087679 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="975ee0d1-2961-44a3-83ea-a6f511831404" containerName="console" Sep 16 13:45:52.088136 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.087778 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b27a2f2d-160e-4534-bb4e-77eff0bf239e" containerName="console" Sep 16 13:45:52.088136 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.087795 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="28371faa-0a08-49cf-8a63-9abe68aca3bc" containerName="registry" Sep 16 13:45:52.088136 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.087804 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="975ee0d1-2961-44a3-83ea-a6f511831404" containerName="console" Sep 16 13:45:52.092478 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.092460 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.102787 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.102369 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f745c9f75-szlvf"] Sep 16 13:45:52.144572 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.144550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-oauth-serving-cert\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.144688 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.144580 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-trusted-ca-bundle\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.144688 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.144611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-config\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.144808 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.144763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-oauth-config\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.144808 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.144802 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-serving-cert\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.144914 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.144836 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-service-ca\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.144914 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.144877 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m842\" (UniqueName: \"kubernetes.io/projected/adca585f-b9eb-42b7-a1aa-430fd6940a58-kube-api-access-5m842\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.245859 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.245831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m842\" (UniqueName: \"kubernetes.io/projected/adca585f-b9eb-42b7-a1aa-430fd6940a58-kube-api-access-5m842\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.245982 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.245870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-oauth-serving-cert\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.245982 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.245887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-trusted-ca-bundle\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.245982 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.245916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-config\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.245982 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.245966 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-oauth-config\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.246229 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.246001 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-serving-cert\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.246229 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.246055 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-service-ca\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.246747 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.246715 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-service-ca\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.246747 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.246723 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-oauth-serving-cert\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.246934 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.246797 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-trusted-ca-bundle\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.246934 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.246824 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-config\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.248449 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.248432 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-serving-cert\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.248503 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.248477 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-oauth-config\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.255579 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.255560 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m842\" (UniqueName: \"kubernetes.io/projected/adca585f-b9eb-42b7-a1aa-430fd6940a58-kube-api-access-5m842\") pod \"console-5f745c9f75-szlvf\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.404586 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.404510 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:45:52.682256 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:52.682229 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f745c9f75-szlvf"] Sep 16 13:45:52.684753 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:45:52.684726 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadca585f_b9eb_42b7_a1aa_430fd6940a58.slice/crio-d15e06d47c410f8e586f1d0054d415478b6e79db055484d1feadd79eb320b09d WatchSource:0}: Error finding container d15e06d47c410f8e586f1d0054d415478b6e79db055484d1feadd79eb320b09d: Status 404 returned error can't find the container with id d15e06d47c410f8e586f1d0054d415478b6e79db055484d1feadd79eb320b09d Sep 16 13:45:53.419420 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:53.419383 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f745c9f75-szlvf" event={"ID":"adca585f-b9eb-42b7-a1aa-430fd6940a58","Type":"ContainerStarted","Data":"3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8"} Sep 16 13:45:53.419799 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:53.419431 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f745c9f75-szlvf" event={"ID":"adca585f-b9eb-42b7-a1aa-430fd6940a58","Type":"ContainerStarted","Data":"d15e06d47c410f8e586f1d0054d415478b6e79db055484d1feadd79eb320b09d"} Sep 16 13:45:53.442690 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:45:53.442646 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f745c9f75-szlvf" podStartSLOduration=1.4426305369999999 podStartE2EDuration="1.442630537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:53.441450607 +0000 UTC m=+187.404809433" watchObservedRunningTime="2026-09-16 13:45:53.442630537 +0000 UTC m=+187.405989364" Sep 16 13:46:02.405672 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:02.405631 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:46:02.406161 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:02.405693 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:46:02.410253 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:02.410229 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:46:02.458261 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:02.458230 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:46:02.516219 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:02.516183 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d8566bd5-56jpv"] Sep 16 13:46:22.686105 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.686072 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg"] Sep 16 13:46:22.689529 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.689511 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:22.692173 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.692132 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:46:22.692173 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.692157 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:46:22.693382 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.693364 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h26rk\"" Sep 16 13:46:22.698557 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.698516 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg"] Sep 16 13:46:22.785234 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.785212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:22.785362 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.785254 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:22.785362 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.785285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmqwv\" (UniqueName: \"kubernetes.io/projected/952d3348-fe54-40ca-80be-cc2c0629dec9-kube-api-access-xmqwv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:22.885872 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.885848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:22.885981 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.885879 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmqwv\" (UniqueName: \"kubernetes.io/projected/952d3348-fe54-40ca-80be-cc2c0629dec9-kube-api-access-xmqwv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:22.886043 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.886018 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:22.886301 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.886279 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:22.886367 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.886333 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:22.896349 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.896324 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmqwv\" (UniqueName: \"kubernetes.io/projected/952d3348-fe54-40ca-80be-cc2c0629dec9-kube-api-access-xmqwv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:22.999359 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:22.999338 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:23.273408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:23.273384 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg"] Sep 16 13:46:23.275773 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:46:23.275739 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952d3348_fe54_40ca_80be_cc2c0629dec9.slice/crio-621a76868bf30e8d24dc0389179216077e0872220c845478fce2c371f5397adf WatchSource:0}: Error finding container 621a76868bf30e8d24dc0389179216077e0872220c845478fce2c371f5397adf: Status 404 returned error can't find the container with id 621a76868bf30e8d24dc0389179216077e0872220c845478fce2c371f5397adf Sep 16 13:46:23.537211 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:23.537127 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" event={"ID":"952d3348-fe54-40ca-80be-cc2c0629dec9","Type":"ContainerStarted","Data":"621a76868bf30e8d24dc0389179216077e0872220c845478fce2c371f5397adf"} Sep 16 13:46:27.538403 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:27.538356 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8d8566bd5-56jpv" podUID="002b49a4-3115-46ca-a3fb-4f3062965de3" containerName="console" containerID="cri-o://20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d" gracePeriod=15 Sep 16 13:46:29.354157 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.354123 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:46:29.429068 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429045 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-serving-cert\") pod \"002b49a4-3115-46ca-a3fb-4f3062965de3\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " Sep 16 13:46:29.429186 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429080 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7rpk\" (UniqueName: \"kubernetes.io/projected/002b49a4-3115-46ca-a3fb-4f3062965de3-kube-api-access-m7rpk\") pod \"002b49a4-3115-46ca-a3fb-4f3062965de3\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " Sep 16 13:46:29.429186 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429105 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-console-config\") pod \"002b49a4-3115-46ca-a3fb-4f3062965de3\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " Sep 16 13:46:29.429291 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429195 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-oauth-config\") pod \"002b49a4-3115-46ca-a3fb-4f3062965de3\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " Sep 16 13:46:29.429291 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429228 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-service-ca\") pod \"002b49a4-3115-46ca-a3fb-4f3062965de3\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " Sep 16 13:46:29.429397 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429296 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-trusted-ca-bundle\") pod \"002b49a4-3115-46ca-a3fb-4f3062965de3\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " Sep 16 13:46:29.429397 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429334 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-oauth-serving-cert\") pod \"002b49a4-3115-46ca-a3fb-4f3062965de3\" (UID: \"002b49a4-3115-46ca-a3fb-4f3062965de3\") " Sep 16 13:46:29.429635 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429584 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-console-config" (OuterVolumeSpecName: "console-config") pod "002b49a4-3115-46ca-a3fb-4f3062965de3" (UID: "002b49a4-3115-46ca-a3fb-4f3062965de3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:29.429715 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429656 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-service-ca" (OuterVolumeSpecName: "service-ca") pod "002b49a4-3115-46ca-a3fb-4f3062965de3" (UID: "002b49a4-3115-46ca-a3fb-4f3062965de3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:29.429829 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429781 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "002b49a4-3115-46ca-a3fb-4f3062965de3" (UID: "002b49a4-3115-46ca-a3fb-4f3062965de3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:29.429885 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.429862 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "002b49a4-3115-46ca-a3fb-4f3062965de3" (UID: "002b49a4-3115-46ca-a3fb-4f3062965de3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:29.431057 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.431034 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "002b49a4-3115-46ca-a3fb-4f3062965de3" (UID: "002b49a4-3115-46ca-a3fb-4f3062965de3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:46:29.431231 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.431118 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "002b49a4-3115-46ca-a3fb-4f3062965de3" (UID: "002b49a4-3115-46ca-a3fb-4f3062965de3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:46:29.431231 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.431168 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002b49a4-3115-46ca-a3fb-4f3062965de3-kube-api-access-m7rpk" (OuterVolumeSpecName: "kube-api-access-m7rpk") pod "002b49a4-3115-46ca-a3fb-4f3062965de3" (UID: "002b49a4-3115-46ca-a3fb-4f3062965de3"). InnerVolumeSpecName "kube-api-access-m7rpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:29.530790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.530770 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-oauth-config\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:46:29.530790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.530789 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:46:29.530956 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.530799 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-trusted-ca-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:46:29.530956 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.530808 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-oauth-serving-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:46:29.530956 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.530818 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/002b49a4-3115-46ca-a3fb-4f3062965de3-console-serving-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:46:29.530956 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.530826 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7rpk\" (UniqueName: \"kubernetes.io/projected/002b49a4-3115-46ca-a3fb-4f3062965de3-kube-api-access-m7rpk\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:46:29.530956 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.530834 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/002b49a4-3115-46ca-a3fb-4f3062965de3-console-config\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:46:29.562575 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.562549 2581 generic.go:356] "Generic (PLEG): container finished" podID="952d3348-fe54-40ca-80be-cc2c0629dec9" containerID="e664e504204b8c4991613b7c8810b2e9459b1c85379b386eb4d897dbc96210db" exitCode=0 Sep 16 13:46:29.562670 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.562635 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" event={"ID":"952d3348-fe54-40ca-80be-cc2c0629dec9","Type":"ContainerDied","Data":"e664e504204b8c4991613b7c8810b2e9459b1c85379b386eb4d897dbc96210db"} Sep 16 13:46:29.564028 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.564002 2581 generic.go:356] "Generic (PLEG): container finished" podID="002b49a4-3115-46ca-a3fb-4f3062965de3" containerID="20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d" exitCode=0 Sep 16 13:46:29.564126 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.564037 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d8566bd5-56jpv" Sep 16 13:46:29.564126 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.564069 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d8566bd5-56jpv" event={"ID":"002b49a4-3115-46ca-a3fb-4f3062965de3","Type":"ContainerDied","Data":"20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d"} Sep 16 13:46:29.564126 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.564119 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d8566bd5-56jpv" event={"ID":"002b49a4-3115-46ca-a3fb-4f3062965de3","Type":"ContainerDied","Data":"74064a6d5cf710179ed95fac544cb9bd386a4eafb0b1919e960ed052662cde1e"} Sep 16 13:46:29.564360 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.564166 2581 scope.go:117] "RemoveContainer" containerID="20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d" Sep 16 13:46:29.577932 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.577909 2581 scope.go:117] "RemoveContainer" containerID="20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d" Sep 16 13:46:29.578231 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:46:29.578203 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d\": container with ID starting with 20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d not found: ID does not exist" containerID="20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d" Sep 16 13:46:29.578290 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.578240 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d"} err="failed to get container status \"20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d\": rpc error: code = NotFound desc = could not find container \"20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d\": container with ID starting with 20608d2f77acaf9c57ed3e065d283905099f84647711a6557b6e25da68eab85d not found: ID does not exist" Sep 16 13:46:29.597202 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.597180 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d8566bd5-56jpv"] Sep 16 13:46:29.602788 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:29.602764 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8d8566bd5-56jpv"] Sep 16 13:46:30.514495 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:30.514463 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="002b49a4-3115-46ca-a3fb-4f3062965de3" path="/var/lib/kubelet/pods/002b49a4-3115-46ca-a3fb-4f3062965de3/volumes" Sep 16 13:46:40.609110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:40.609073 2581 generic.go:356] "Generic (PLEG): container finished" podID="952d3348-fe54-40ca-80be-cc2c0629dec9" containerID="155e5969283f5a91ed4c8d9500c2021c9691508f235d1d6dead9741d65ae44c7" exitCode=0 Sep 16 13:46:40.609535 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:40.609165 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" event={"ID":"952d3348-fe54-40ca-80be-cc2c0629dec9","Type":"ContainerDied","Data":"155e5969283f5a91ed4c8d9500c2021c9691508f235d1d6dead9741d65ae44c7"} Sep 16 13:46:47.641199 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:47.641163 2581 generic.go:356] "Generic (PLEG): container finished" podID="952d3348-fe54-40ca-80be-cc2c0629dec9" containerID="5298960e4b47584e146a9b0a8dac78fb3366d3b908eb8114b25e596620bf1820" exitCode=0 Sep 16 13:46:47.641580 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:47.641239 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" event={"ID":"952d3348-fe54-40ca-80be-cc2c0629dec9","Type":"ContainerDied","Data":"5298960e4b47584e146a9b0a8dac78fb3366d3b908eb8114b25e596620bf1820"} Sep 16 13:46:48.768469 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:48.768444 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:48.888087 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:48.888057 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-bundle\") pod \"952d3348-fe54-40ca-80be-cc2c0629dec9\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " Sep 16 13:46:48.888250 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:48.888110 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-util\") pod \"952d3348-fe54-40ca-80be-cc2c0629dec9\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " Sep 16 13:46:48.888250 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:48.888134 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmqwv\" (UniqueName: \"kubernetes.io/projected/952d3348-fe54-40ca-80be-cc2c0629dec9-kube-api-access-xmqwv\") pod \"952d3348-fe54-40ca-80be-cc2c0629dec9\" (UID: \"952d3348-fe54-40ca-80be-cc2c0629dec9\") " Sep 16 13:46:48.888752 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:48.888723 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-bundle" (OuterVolumeSpecName: "bundle") pod "952d3348-fe54-40ca-80be-cc2c0629dec9" (UID: "952d3348-fe54-40ca-80be-cc2c0629dec9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:48.890400 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:48.890376 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952d3348-fe54-40ca-80be-cc2c0629dec9-kube-api-access-xmqwv" (OuterVolumeSpecName: "kube-api-access-xmqwv") pod "952d3348-fe54-40ca-80be-cc2c0629dec9" (UID: "952d3348-fe54-40ca-80be-cc2c0629dec9"). InnerVolumeSpecName "kube-api-access-xmqwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:48.892287 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:48.892260 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-util" (OuterVolumeSpecName: "util") pod "952d3348-fe54-40ca-80be-cc2c0629dec9" (UID: "952d3348-fe54-40ca-80be-cc2c0629dec9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:46:48.988807 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:48.988784 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-util\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:46:48.988807 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:48.988807 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmqwv\" (UniqueName: \"kubernetes.io/projected/952d3348-fe54-40ca-80be-cc2c0629dec9-kube-api-access-xmqwv\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:46:48.988953 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:48.988818 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/952d3348-fe54-40ca-80be-cc2c0629dec9-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:46:49.650998 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:49.650972 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" Sep 16 13:46:49.651188 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:49.650973 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qkrg" event={"ID":"952d3348-fe54-40ca-80be-cc2c0629dec9","Type":"ContainerDied","Data":"621a76868bf30e8d24dc0389179216077e0872220c845478fce2c371f5397adf"} Sep 16 13:46:49.651188 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:49.651079 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="621a76868bf30e8d24dc0389179216077e0872220c845478fce2c371f5397adf" Sep 16 13:46:55.996430 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996397 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf"] Sep 16 13:46:55.996909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996670 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="952d3348-fe54-40ca-80be-cc2c0629dec9" containerName="util" Sep 16 13:46:55.996909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996680 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="952d3348-fe54-40ca-80be-cc2c0629dec9" containerName="util" Sep 16 13:46:55.996909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996695 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="002b49a4-3115-46ca-a3fb-4f3062965de3" containerName="console" Sep 16 13:46:55.996909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996701 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="002b49a4-3115-46ca-a3fb-4f3062965de3" containerName="console" Sep 16 13:46:55.996909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996710 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="952d3348-fe54-40ca-80be-cc2c0629dec9" containerName="pull" Sep 16 13:46:55.996909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996715 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="952d3348-fe54-40ca-80be-cc2c0629dec9" containerName="pull" Sep 16 13:46:55.996909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996728 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="952d3348-fe54-40ca-80be-cc2c0629dec9" containerName="extract" Sep 16 13:46:55.996909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996733 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="952d3348-fe54-40ca-80be-cc2c0629dec9" containerName="extract" Sep 16 13:46:55.996909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996785 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="002b49a4-3115-46ca-a3fb-4f3062965de3" containerName="console" Sep 16 13:46:55.996909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.996792 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="952d3348-fe54-40ca-80be-cc2c0629dec9" containerName="extract" Sep 16 13:46:55.999902 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:55.999885 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" Sep 16 13:46:56.002750 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.002731 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 16 13:46:56.003328 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.003310 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:46:56.003416 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.003399 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fvrqn\"" Sep 16 13:46:56.019798 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.019777 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf"] Sep 16 13:46:56.150780 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.150745 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfzp\" (UniqueName: \"kubernetes.io/projected/1ffcb868-ffd4-471c-8aa4-2673e28ae73c-kube-api-access-krfzp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gbpjf\" (UID: \"1ffcb868-ffd4-471c-8aa4-2673e28ae73c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" Sep 16 13:46:56.150924 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.150817 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ffcb868-ffd4-471c-8aa4-2673e28ae73c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gbpjf\" (UID: \"1ffcb868-ffd4-471c-8aa4-2673e28ae73c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" Sep 16 13:46:56.251813 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.251744 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ffcb868-ffd4-471c-8aa4-2673e28ae73c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gbpjf\" (UID: \"1ffcb868-ffd4-471c-8aa4-2673e28ae73c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" Sep 16 13:46:56.251813 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.251793 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfzp\" (UniqueName: \"kubernetes.io/projected/1ffcb868-ffd4-471c-8aa4-2673e28ae73c-kube-api-access-krfzp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gbpjf\" (UID: \"1ffcb868-ffd4-471c-8aa4-2673e28ae73c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" Sep 16 13:46:56.252105 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.252088 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ffcb868-ffd4-471c-8aa4-2673e28ae73c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gbpjf\" (UID: \"1ffcb868-ffd4-471c-8aa4-2673e28ae73c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" Sep 16 13:46:56.260680 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.260660 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfzp\" (UniqueName: \"kubernetes.io/projected/1ffcb868-ffd4-471c-8aa4-2673e28ae73c-kube-api-access-krfzp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gbpjf\" (UID: \"1ffcb868-ffd4-471c-8aa4-2673e28ae73c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" Sep 16 13:46:56.308611 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.308593 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" Sep 16 13:46:56.589120 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.589090 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf"] Sep 16 13:46:56.590827 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:46:56.590802 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffcb868_ffd4_471c_8aa4_2673e28ae73c.slice/crio-33ddb8828bebe54331e4ba75751abe4872e418f14a14cd8d66655d12c5fad550 WatchSource:0}: Error finding container 33ddb8828bebe54331e4ba75751abe4872e418f14a14cd8d66655d12c5fad550: Status 404 returned error can't find the container with id 33ddb8828bebe54331e4ba75751abe4872e418f14a14cd8d66655d12c5fad550 Sep 16 13:46:56.678344 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:56.678314 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" event={"ID":"1ffcb868-ffd4-471c-8aa4-2673e28ae73c","Type":"ContainerStarted","Data":"33ddb8828bebe54331e4ba75751abe4872e418f14a14cd8d66655d12c5fad550"} Sep 16 13:46:58.688243 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:58.688198 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" event={"ID":"1ffcb868-ffd4-471c-8aa4-2673e28ae73c","Type":"ContainerStarted","Data":"c1d0ac88603da2116101e7f04a8d8431e2b629666856bb694ca39b4f07e42929"} Sep 16 13:46:58.714856 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:58.714808 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gbpjf" podStartSLOduration=2.173637851 podStartE2EDuration="3.714792996s" totalImagesPullingTime="1.541155145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:58.712718717 +0000 UTC m=+252.676077544" watchObservedRunningTime="2026-09-16 13:46:58.714792996 +0000 UTC m=+252.678151822" Sep 16 13:46:59.955175 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:59.955126 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-46kkk"] Sep 16 13:46:59.958421 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:59.958401 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" Sep 16 13:46:59.960954 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:59.960934 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 13:46:59.961048 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:59.960938 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 13:46:59.962062 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:59.962044 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-72s9n\"" Sep 16 13:46:59.966884 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:46:59.966866 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-46kkk"] Sep 16 13:47:00.082193 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.082137 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psr9w\" (UniqueName: \"kubernetes.io/projected/663cd88d-6f97-4388-936e-81f3f88f7680-kube-api-access-psr9w\") pod \"cert-manager-webhook-6486ffd997-46kkk\" (UID: \"663cd88d-6f97-4388-936e-81f3f88f7680\") " pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" Sep 16 13:47:00.082353 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.082199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/663cd88d-6f97-4388-936e-81f3f88f7680-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-46kkk\" (UID: \"663cd88d-6f97-4388-936e-81f3f88f7680\") " pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" Sep 16 13:47:00.183133 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.183090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psr9w\" (UniqueName: \"kubernetes.io/projected/663cd88d-6f97-4388-936e-81f3f88f7680-kube-api-access-psr9w\") pod \"cert-manager-webhook-6486ffd997-46kkk\" (UID: \"663cd88d-6f97-4388-936e-81f3f88f7680\") " pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" Sep 16 13:47:00.183293 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.183163 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/663cd88d-6f97-4388-936e-81f3f88f7680-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-46kkk\" (UID: \"663cd88d-6f97-4388-936e-81f3f88f7680\") " pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" Sep 16 13:47:00.192813 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.192780 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/663cd88d-6f97-4388-936e-81f3f88f7680-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-46kkk\" (UID: \"663cd88d-6f97-4388-936e-81f3f88f7680\") " pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" Sep 16 13:47:00.193008 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.192859 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psr9w\" (UniqueName: \"kubernetes.io/projected/663cd88d-6f97-4388-936e-81f3f88f7680-kube-api-access-psr9w\") pod \"cert-manager-webhook-6486ffd997-46kkk\" (UID: \"663cd88d-6f97-4388-936e-81f3f88f7680\") " pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" Sep 16 13:47:00.276308 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.276262 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" Sep 16 13:47:00.472697 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.472650 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx"] Sep 16 13:47:00.476244 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.476226 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:00.478862 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.478835 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:47:00.478967 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.478837 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h26rk\"" Sep 16 13:47:00.479875 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.479815 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:47:00.486264 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.486243 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx"] Sep 16 13:47:00.548358 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.548297 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-46kkk"] Sep 16 13:47:00.551348 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:47:00.551324 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod663cd88d_6f97_4388_936e_81f3f88f7680.slice/crio-5465c8953c08a2d44230494e8b8c38b88a31401d5e46148c8e960e84627eaf0b WatchSource:0}: Error finding container 5465c8953c08a2d44230494e8b8c38b88a31401d5e46148c8e960e84627eaf0b: Status 404 returned error can't find the container with id 5465c8953c08a2d44230494e8b8c38b88a31401d5e46148c8e960e84627eaf0b Sep 16 13:47:00.586442 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.586420 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgkdc\" (UniqueName: \"kubernetes.io/projected/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-kube-api-access-mgkdc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:00.586532 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.586484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:00.586532 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.586503 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:00.687554 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.687531 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgkdc\" (UniqueName: \"kubernetes.io/projected/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-kube-api-access-mgkdc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:00.687626 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.687570 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:00.687626 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.687587 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:00.687924 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.687907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:00.687958 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.687912 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:00.697040 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.697017 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgkdc\" (UniqueName: \"kubernetes.io/projected/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-kube-api-access-mgkdc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:00.697722 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.697701 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" event={"ID":"663cd88d-6f97-4388-936e-81f3f88f7680","Type":"ContainerStarted","Data":"5465c8953c08a2d44230494e8b8c38b88a31401d5e46148c8e960e84627eaf0b"} Sep 16 13:47:00.785642 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:00.785620 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:01.054532 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.054505 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx"] Sep 16 13:47:01.056489 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:47:01.056463 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8a5d62_2f6a_442b_b1ff_7bdd1ce80b3e.slice/crio-3eb5cd5d7d4cdf770dcef98e8256e57dd21c9be93ba46c697b3f398fbf2712ef WatchSource:0}: Error finding container 3eb5cd5d7d4cdf770dcef98e8256e57dd21c9be93ba46c697b3f398fbf2712ef: Status 404 returned error can't find the container with id 3eb5cd5d7d4cdf770dcef98e8256e57dd21c9be93ba46c697b3f398fbf2712ef Sep 16 13:47:01.704863 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.704819 2581 generic.go:356] "Generic (PLEG): container finished" podID="1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" containerID="d3444c3a1d7c5996bc32b4664feec5b2f6496689b12b256ce240c1aeeda15388" exitCode=0 Sep 16 13:47:01.705042 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.704912 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" event={"ID":"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e","Type":"ContainerDied","Data":"d3444c3a1d7c5996bc32b4664feec5b2f6496689b12b256ce240c1aeeda15388"} Sep 16 13:47:01.705042 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.704966 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" event={"ID":"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e","Type":"ContainerStarted","Data":"3eb5cd5d7d4cdf770dcef98e8256e57dd21c9be93ba46c697b3f398fbf2712ef"} Sep 16 13:47:01.723580 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.722274 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r2259"] Sep 16 13:47:01.727208 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.727166 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" Sep 16 13:47:01.729858 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.729836 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vfqh9\"" Sep 16 13:47:01.732558 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.732527 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r2259"] Sep 16 13:47:01.796623 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.796585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74df9d66-dfa9-46df-9f55-6a69cffc31d3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r2259\" (UID: \"74df9d66-dfa9-46df-9f55-6a69cffc31d3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" Sep 16 13:47:01.796825 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.796752 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n92s5\" (UniqueName: \"kubernetes.io/projected/74df9d66-dfa9-46df-9f55-6a69cffc31d3-kube-api-access-n92s5\") pod \"cert-manager-cainjector-559cdd8c87-r2259\" (UID: \"74df9d66-dfa9-46df-9f55-6a69cffc31d3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" Sep 16 13:47:01.898011 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.897971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74df9d66-dfa9-46df-9f55-6a69cffc31d3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r2259\" (UID: \"74df9d66-dfa9-46df-9f55-6a69cffc31d3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" Sep 16 13:47:01.898231 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.898082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n92s5\" (UniqueName: \"kubernetes.io/projected/74df9d66-dfa9-46df-9f55-6a69cffc31d3-kube-api-access-n92s5\") pod \"cert-manager-cainjector-559cdd8c87-r2259\" (UID: \"74df9d66-dfa9-46df-9f55-6a69cffc31d3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" Sep 16 13:47:01.907872 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.907844 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74df9d66-dfa9-46df-9f55-6a69cffc31d3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r2259\" (UID: \"74df9d66-dfa9-46df-9f55-6a69cffc31d3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" Sep 16 13:47:01.908405 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:01.908367 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n92s5\" (UniqueName: \"kubernetes.io/projected/74df9d66-dfa9-46df-9f55-6a69cffc31d3-kube-api-access-n92s5\") pod \"cert-manager-cainjector-559cdd8c87-r2259\" (UID: \"74df9d66-dfa9-46df-9f55-6a69cffc31d3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" Sep 16 13:47:02.039667 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:02.039629 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" Sep 16 13:47:02.319804 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:02.319778 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r2259"] Sep 16 13:47:03.151515 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:47:03.151484 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74df9d66_dfa9_46df_9f55_6a69cffc31d3.slice/crio-68b1a4f36d9b22ed377a8ca5ffa7a0b3f49fc6ab54c3171824db017f6f4a4cf9 WatchSource:0}: Error finding container 68b1a4f36d9b22ed377a8ca5ffa7a0b3f49fc6ab54c3171824db017f6f4a4cf9: Status 404 returned error can't find the container with id 68b1a4f36d9b22ed377a8ca5ffa7a0b3f49fc6ab54c3171824db017f6f4a4cf9 Sep 16 13:47:03.717734 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:03.717441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" event={"ID":"663cd88d-6f97-4388-936e-81f3f88f7680","Type":"ContainerStarted","Data":"fb8d88a07beca7d4ecc67e8bb283cde0517926d11d175cda2063562dd28e0f6d"} Sep 16 13:47:03.717734 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:03.717603 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" Sep 16 13:47:03.721233 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:03.721206 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" event={"ID":"74df9d66-dfa9-46df-9f55-6a69cffc31d3","Type":"ContainerStarted","Data":"00f34292bb12253b1f6e12c7a3b9421eeb511c4075b6b1761e4e145162812060"} Sep 16 13:47:03.721364 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:03.721245 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" event={"ID":"74df9d66-dfa9-46df-9f55-6a69cffc31d3","Type":"ContainerStarted","Data":"68b1a4f36d9b22ed377a8ca5ffa7a0b3f49fc6ab54c3171824db017f6f4a4cf9"} Sep 16 13:47:03.735862 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:03.735820 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" podStartSLOduration=2.079323959 podStartE2EDuration="4.73580656s" totalImagesPullingTime="2.656482601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:03.734329696 +0000 UTC m=+257.697688524" watchObservedRunningTime="2026-09-16 13:47:03.73580656 +0000 UTC m=+257.699165388" Sep 16 13:47:03.751835 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:03.751703 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r2259" podStartSLOduration=2.393266763 podStartE2EDuration="2.751684217s" totalImagesPullingTime="358.417454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:03.749761796 +0000 UTC m=+257.713120653" watchObservedRunningTime="2026-09-16 13:47:03.751684217 +0000 UTC m=+257.715043059" Sep 16 13:47:05.732277 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:05.732244 2581 generic.go:356] "Generic (PLEG): container finished" podID="1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" containerID="327c345579d976b939a0f6c2c5bd820bec9240f8e3b91ed0abc6bb564d14f735" exitCode=0 Sep 16 13:47:05.732631 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:05.732316 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" event={"ID":"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e","Type":"ContainerDied","Data":"327c345579d976b939a0f6c2c5bd820bec9240f8e3b91ed0abc6bb564d14f735"} Sep 16 13:47:06.738807 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:06.738767 2581 generic.go:356] "Generic (PLEG): container finished" podID="1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" containerID="27f7052564e2db6e46bb92587ffc5c7784e5411e3faf7ff6f16283bc9994b052" exitCode=0 Sep 16 13:47:06.739220 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:06.738848 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" event={"ID":"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e","Type":"ContainerDied","Data":"27f7052564e2db6e46bb92587ffc5c7784e5411e3faf7ff6f16283bc9994b052"} Sep 16 13:47:07.869477 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:07.869452 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:07.953283 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:07.953255 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-util\") pod \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " Sep 16 13:47:07.953414 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:07.953297 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgkdc\" (UniqueName: \"kubernetes.io/projected/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-kube-api-access-mgkdc\") pod \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " Sep 16 13:47:07.953414 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:07.953347 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-bundle\") pod \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\" (UID: \"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e\") " Sep 16 13:47:07.953759 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:07.953735 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-bundle" (OuterVolumeSpecName: "bundle") pod "1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" (UID: "1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:07.955320 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:07.955299 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-kube-api-access-mgkdc" (OuterVolumeSpecName: "kube-api-access-mgkdc") pod "1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" (UID: "1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e"). InnerVolumeSpecName "kube-api-access-mgkdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:47:07.959451 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:07.959423 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-util" (OuterVolumeSpecName: "util") pod "1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" (UID: "1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:08.055216 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:08.055127 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-util\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:47:08.055216 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:08.055175 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgkdc\" (UniqueName: \"kubernetes.io/projected/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-kube-api-access-mgkdc\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:47:08.055216 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:08.055186 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:47:08.749839 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:08.749801 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" Sep 16 13:47:08.749993 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:08.749807 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmgdxx" event={"ID":"1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e","Type":"ContainerDied","Data":"3eb5cd5d7d4cdf770dcef98e8256e57dd21c9be93ba46c697b3f398fbf2712ef"} Sep 16 13:47:08.749993 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:08.749961 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb5cd5d7d4cdf770dcef98e8256e57dd21c9be93ba46c697b3f398fbf2712ef" Sep 16 13:47:09.728716 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:09.728646 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-46kkk" Sep 16 13:47:17.119469 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.119438 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg"] Sep 16 13:47:17.119898 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.119704 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" containerName="pull" Sep 16 13:47:17.119898 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.119714 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" containerName="pull" Sep 16 13:47:17.119898 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.119730 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" containerName="util" Sep 16 13:47:17.119898 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.119735 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" containerName="util" Sep 16 13:47:17.119898 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.119745 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" containerName="extract" Sep 16 13:47:17.119898 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.119752 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" containerName="extract" Sep 16 13:47:17.119898 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.119804 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8a5d62-2f6a-442b-b1ff-7bdd1ce80b3e" containerName="extract" Sep 16 13:47:17.126448 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.126428 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" Sep 16 13:47:17.129588 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.129563 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-bwqvp\"" Sep 16 13:47:17.129672 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.129633 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 13:47:17.130422 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.130405 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:47:17.134181 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.134157 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg"] Sep 16 13:47:17.227078 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.227047 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/939c43b1-48b9-4d93-977c-21d5e91e6483-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hdwlg\" (UID: \"939c43b1-48b9-4d93-977c-21d5e91e6483\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" Sep 16 13:47:17.227078 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.227078 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pz78\" (UniqueName: \"kubernetes.io/projected/939c43b1-48b9-4d93-977c-21d5e91e6483-kube-api-access-2pz78\") pod \"openshift-lws-operator-fd8ccff4c-hdwlg\" (UID: \"939c43b1-48b9-4d93-977c-21d5e91e6483\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" Sep 16 13:47:17.327999 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.327961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/939c43b1-48b9-4d93-977c-21d5e91e6483-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hdwlg\" (UID: \"939c43b1-48b9-4d93-977c-21d5e91e6483\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" Sep 16 13:47:17.327999 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.328000 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pz78\" (UniqueName: \"kubernetes.io/projected/939c43b1-48b9-4d93-977c-21d5e91e6483-kube-api-access-2pz78\") pod \"openshift-lws-operator-fd8ccff4c-hdwlg\" (UID: \"939c43b1-48b9-4d93-977c-21d5e91e6483\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" Sep 16 13:47:17.328380 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.328354 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/939c43b1-48b9-4d93-977c-21d5e91e6483-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hdwlg\" (UID: \"939c43b1-48b9-4d93-977c-21d5e91e6483\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" Sep 16 13:47:17.346156 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.346114 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pz78\" (UniqueName: \"kubernetes.io/projected/939c43b1-48b9-4d93-977c-21d5e91e6483-kube-api-access-2pz78\") pod \"openshift-lws-operator-fd8ccff4c-hdwlg\" (UID: \"939c43b1-48b9-4d93-977c-21d5e91e6483\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" Sep 16 13:47:17.436774 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.436698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" Sep 16 13:47:17.708647 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.708622 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg"] Sep 16 13:47:17.710890 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:47:17.710865 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod939c43b1_48b9_4d93_977c_21d5e91e6483.slice/crio-10d798ebbcdd9ab4f3c632c48b53ec05c82ce51a00001634abe8c8e68b6482c4 WatchSource:0}: Error finding container 10d798ebbcdd9ab4f3c632c48b53ec05c82ce51a00001634abe8c8e68b6482c4: Status 404 returned error can't find the container with id 10d798ebbcdd9ab4f3c632c48b53ec05c82ce51a00001634abe8c8e68b6482c4 Sep 16 13:47:17.791509 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:17.791475 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" event={"ID":"939c43b1-48b9-4d93-977c-21d5e91e6483","Type":"ContainerStarted","Data":"10d798ebbcdd9ab4f3c632c48b53ec05c82ce51a00001634abe8c8e68b6482c4"} Sep 16 13:47:20.807114 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:20.807076 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" event={"ID":"939c43b1-48b9-4d93-977c-21d5e91e6483","Type":"ContainerStarted","Data":"84b890f7659efb736c76caa403066656dbc4eef413a6f98bc0094de5f96e9efb"} Sep 16 13:47:20.824776 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:20.824729 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hdwlg" podStartSLOduration=1.728709082 podStartE2EDuration="3.824714924s" totalImagesPullingTime="2.096005842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:20.823885126 +0000 UTC m=+274.787243955" watchObservedRunningTime="2026-09-16 13:47:20.824714924 +0000 UTC m=+274.788073751" Sep 16 13:47:24.235380 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.235349 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn"] Sep 16 13:47:24.238825 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.238808 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.241439 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.241415 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:47:24.242282 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.242265 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:47:24.242282 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.242275 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h26rk\"" Sep 16 13:47:24.247465 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.247443 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn"] Sep 16 13:47:24.285274 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.285216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.285274 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.285248 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwhf\" (UniqueName: \"kubernetes.io/projected/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-kube-api-access-szwhf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.285437 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.285278 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.386695 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.386665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.386823 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.386701 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwhf\" (UniqueName: \"kubernetes.io/projected/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-kube-api-access-szwhf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.386823 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.386742 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.387019 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.386997 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.387087 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.387072 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.395561 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.395531 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwhf\" (UniqueName: \"kubernetes.io/projected/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-kube-api-access-szwhf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.548589 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.548509 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:24.819836 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:24.819766 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn"] Sep 16 13:47:24.823096 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:47:24.823065 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81fae4a_0a6c_4f0b_8984_2f8bc31c519b.slice/crio-4c4bdfa9835ed707f9db77b85db55adbaa351e5207d97eb5b5bc266cf6b9a291 WatchSource:0}: Error finding container 4c4bdfa9835ed707f9db77b85db55adbaa351e5207d97eb5b5bc266cf6b9a291: Status 404 returned error can't find the container with id 4c4bdfa9835ed707f9db77b85db55adbaa351e5207d97eb5b5bc266cf6b9a291 Sep 16 13:47:25.832021 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:25.831985 2581 generic.go:356] "Generic (PLEG): container finished" podID="e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" containerID="bd5032bfd764267ed272e44901513091ba3631c967070e017b805736c011fc5d" exitCode=0 Sep 16 13:47:25.832420 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:25.832075 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" event={"ID":"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b","Type":"ContainerDied","Data":"bd5032bfd764267ed272e44901513091ba3631c967070e017b805736c011fc5d"} Sep 16 13:47:25.832420 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:25.832180 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" event={"ID":"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b","Type":"ContainerStarted","Data":"4c4bdfa9835ed707f9db77b85db55adbaa351e5207d97eb5b5bc266cf6b9a291"} Sep 16 13:47:26.838514 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:26.838481 2581 generic.go:356] "Generic (PLEG): container finished" podID="e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" containerID="8ff40e607178f601e4dda1343db003a48f80398e896c96b196fc652698f45ed2" exitCode=0 Sep 16 13:47:26.838944 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:26.838564 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" event={"ID":"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b","Type":"ContainerDied","Data":"8ff40e607178f601e4dda1343db003a48f80398e896c96b196fc652698f45ed2"} Sep 16 13:47:27.844732 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:27.844696 2581 generic.go:356] "Generic (PLEG): container finished" podID="e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" containerID="4ffef156d7aea20a4c8fa8c2a1ab64cb89eb7c8cd5ec467b4de0ba61a7157e4a" exitCode=0 Sep 16 13:47:27.845106 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:27.844738 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" event={"ID":"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b","Type":"ContainerDied","Data":"4ffef156d7aea20a4c8fa8c2a1ab64cb89eb7c8cd5ec467b4de0ba61a7157e4a"} Sep 16 13:47:28.972106 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:28.972082 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:29.022835 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.022799 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szwhf\" (UniqueName: \"kubernetes.io/projected/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-kube-api-access-szwhf\") pod \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " Sep 16 13:47:29.022977 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.022879 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-bundle\") pod \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " Sep 16 13:47:29.022977 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.022918 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-util\") pod \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\" (UID: \"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b\") " Sep 16 13:47:29.023745 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.023714 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-bundle" (OuterVolumeSpecName: "bundle") pod "e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" (UID: "e81fae4a-0a6c-4f0b-8984-2f8bc31c519b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:29.024874 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.024853 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-kube-api-access-szwhf" (OuterVolumeSpecName: "kube-api-access-szwhf") pod "e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" (UID: "e81fae4a-0a6c-4f0b-8984-2f8bc31c519b"). InnerVolumeSpecName "kube-api-access-szwhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:47:29.031026 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.031000 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-util" (OuterVolumeSpecName: "util") pod "e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" (UID: "e81fae4a-0a6c-4f0b-8984-2f8bc31c519b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:29.123952 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.123888 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwhf\" (UniqueName: \"kubernetes.io/projected/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-kube-api-access-szwhf\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:47:29.123952 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.123912 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:47:29.123952 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.123922 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e81fae4a-0a6c-4f0b-8984-2f8bc31c519b-util\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:47:29.855742 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.855715 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" Sep 16 13:47:29.855742 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.855736 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lwfn" event={"ID":"e81fae4a-0a6c-4f0b-8984-2f8bc31c519b","Type":"ContainerDied","Data":"4c4bdfa9835ed707f9db77b85db55adbaa351e5207d97eb5b5bc266cf6b9a291"} Sep 16 13:47:29.855940 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:29.855775 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c4bdfa9835ed707f9db77b85db55adbaa351e5207d97eb5b5bc266cf6b9a291" Sep 16 13:47:34.866464 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.866428 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7"] Sep 16 13:47:34.866818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.866731 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" containerName="util" Sep 16 13:47:34.866818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.866742 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" containerName="util" Sep 16 13:47:34.866818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.866750 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" containerName="pull" Sep 16 13:47:34.866818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.866755 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" containerName="pull" Sep 16 13:47:34.866818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.866765 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" containerName="extract" Sep 16 13:47:34.866818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.866770 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" containerName="extract" Sep 16 13:47:34.867011 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.866822 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e81fae4a-0a6c-4f0b-8984-2f8bc31c519b" containerName="extract" Sep 16 13:47:34.872678 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.872655 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:34.875421 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.875399 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h26rk\"" Sep 16 13:47:34.875421 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.875411 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:47:34.876244 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.876209 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:47:34.882899 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.882877 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7"] Sep 16 13:47:34.971653 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.971615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:34.971818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.971700 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:34.971818 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:34.971728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qwvx\" (UniqueName: \"kubernetes.io/projected/58c46b50-6396-4568-a0ec-d58d51359df3-kube-api-access-8qwvx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:35.072663 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.072630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwvx\" (UniqueName: \"kubernetes.io/projected/58c46b50-6396-4568-a0ec-d58d51359df3-kube-api-access-8qwvx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:35.072786 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.072684 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:35.072786 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.072745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:35.073152 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.073125 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:35.073200 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.073181 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:35.081759 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.081727 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwvx\" (UniqueName: \"kubernetes.io/projected/58c46b50-6396-4568-a0ec-d58d51359df3-kube-api-access-8qwvx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:35.187482 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.187409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:35.464360 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.464331 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7"] Sep 16 13:47:35.467495 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:47:35.467470 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c46b50_6396_4568_a0ec_d58d51359df3.slice/crio-7ebfc98280007fa6323fc5c5bbf2b9911f953815a6483bdc63a74ac8bac052a6 WatchSource:0}: Error finding container 7ebfc98280007fa6323fc5c5bbf2b9911f953815a6483bdc63a74ac8bac052a6: Status 404 returned error can't find the container with id 7ebfc98280007fa6323fc5c5bbf2b9911f953815a6483bdc63a74ac8bac052a6 Sep 16 13:47:35.892117 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.892025 2581 generic.go:356] "Generic (PLEG): container finished" podID="58c46b50-6396-4568-a0ec-d58d51359df3" containerID="43e9b00d7241f61f73039798fb4c146ce8847a4eb5ce45a657aba91af1ecca62" exitCode=0 Sep 16 13:47:35.892475 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.892108 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" event={"ID":"58c46b50-6396-4568-a0ec-d58d51359df3","Type":"ContainerDied","Data":"43e9b00d7241f61f73039798fb4c146ce8847a4eb5ce45a657aba91af1ecca62"} Sep 16 13:47:35.892475 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:35.892175 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" event={"ID":"58c46b50-6396-4568-a0ec-d58d51359df3","Type":"ContainerStarted","Data":"7ebfc98280007fa6323fc5c5bbf2b9911f953815a6483bdc63a74ac8bac052a6"} Sep 16 13:47:36.948019 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.947983 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4"] Sep 16 13:47:36.952197 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.952180 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:36.954392 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.954371 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 13:47:36.954531 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.954509 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 13:47:36.954965 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.954945 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 13:47:36.955208 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.955192 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 13:47:36.955318 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.955301 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n7f75\"" Sep 16 13:47:36.966951 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.966929 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4"] Sep 16 13:47:36.987400 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.987374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/382e3f4d-0714-4b70-8e27-d471c525de0e-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:36.987694 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.987442 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/382e3f4d-0714-4b70-8e27-d471c525de0e-tmp\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:36.987694 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.987550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/382e3f4d-0714-4b70-8e27-d471c525de0e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:36.987694 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.987600 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/382e3f4d-0714-4b70-8e27-d471c525de0e-manifests\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:36.987694 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:36.987657 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lb2z\" (UniqueName: \"kubernetes.io/projected/382e3f4d-0714-4b70-8e27-d471c525de0e-kube-api-access-9lb2z\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.088624 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.088583 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/382e3f4d-0714-4b70-8e27-d471c525de0e-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.088624 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.088619 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/382e3f4d-0714-4b70-8e27-d471c525de0e-tmp\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.088859 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.088678 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/382e3f4d-0714-4b70-8e27-d471c525de0e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.088859 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.088718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/382e3f4d-0714-4b70-8e27-d471c525de0e-manifests\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.088859 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.088748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lb2z\" (UniqueName: \"kubernetes.io/projected/382e3f4d-0714-4b70-8e27-d471c525de0e-kube-api-access-9lb2z\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.089411 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.089368 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/382e3f4d-0714-4b70-8e27-d471c525de0e-manifests\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.091501 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.091478 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/382e3f4d-0714-4b70-8e27-d471c525de0e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.091597 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.091478 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/382e3f4d-0714-4b70-8e27-d471c525de0e-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.091597 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.091560 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/382e3f4d-0714-4b70-8e27-d471c525de0e-tmp\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.097392 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.097373 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lb2z\" (UniqueName: \"kubernetes.io/projected/382e3f4d-0714-4b70-8e27-d471c525de0e-kube-api-access-9lb2z\") pod \"opendatahub-operator-controller-manager-f5fcfb689-wm9r4\" (UID: \"382e3f4d-0714-4b70-8e27-d471c525de0e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.262246 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.262213 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:37.550753 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.550730 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4"] Sep 16 13:47:37.552792 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:47:37.552765 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382e3f4d_0714_4b70_8e27_d471c525de0e.slice/crio-fc9adb66d2716642115d3a6d7fd6c7af6fb05ec3e1b09dfdaca7f12dd2fd4160 WatchSource:0}: Error finding container fc9adb66d2716642115d3a6d7fd6c7af6fb05ec3e1b09dfdaca7f12dd2fd4160: Status 404 returned error can't find the container with id fc9adb66d2716642115d3a6d7fd6c7af6fb05ec3e1b09dfdaca7f12dd2fd4160 Sep 16 13:47:37.902527 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.902454 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" event={"ID":"382e3f4d-0714-4b70-8e27-d471c525de0e","Type":"ContainerStarted","Data":"fc9adb66d2716642115d3a6d7fd6c7af6fb05ec3e1b09dfdaca7f12dd2fd4160"} Sep 16 13:47:37.904418 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.904390 2581 generic.go:356] "Generic (PLEG): container finished" podID="58c46b50-6396-4568-a0ec-d58d51359df3" containerID="d707ec6454929f5db6b4698753e113db15fb291c98bb7c64693475e491c56ca5" exitCode=0 Sep 16 13:47:37.904523 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:37.904470 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" event={"ID":"58c46b50-6396-4568-a0ec-d58d51359df3","Type":"ContainerDied","Data":"d707ec6454929f5db6b4698753e113db15fb291c98bb7c64693475e491c56ca5"} Sep 16 13:47:38.921490 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:38.921450 2581 generic.go:356] "Generic (PLEG): container finished" podID="58c46b50-6396-4568-a0ec-d58d51359df3" containerID="9b08082619a25bcfd4be882108bfc5858f852d131fc8ee3a75034e15b68d18e0" exitCode=0 Sep 16 13:47:38.921928 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:38.921519 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" event={"ID":"58c46b50-6396-4568-a0ec-d58d51359df3","Type":"ContainerDied","Data":"9b08082619a25bcfd4be882108bfc5858f852d131fc8ee3a75034e15b68d18e0"} Sep 16 13:47:40.455232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.455203 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:40.517561 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.517538 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qwvx\" (UniqueName: \"kubernetes.io/projected/58c46b50-6396-4568-a0ec-d58d51359df3-kube-api-access-8qwvx\") pod \"58c46b50-6396-4568-a0ec-d58d51359df3\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " Sep 16 13:47:40.517652 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.517587 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-util\") pod \"58c46b50-6396-4568-a0ec-d58d51359df3\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " Sep 16 13:47:40.517652 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.517649 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-bundle\") pod \"58c46b50-6396-4568-a0ec-d58d51359df3\" (UID: \"58c46b50-6396-4568-a0ec-d58d51359df3\") " Sep 16 13:47:40.518374 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.518351 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-bundle" (OuterVolumeSpecName: "bundle") pod "58c46b50-6396-4568-a0ec-d58d51359df3" (UID: "58c46b50-6396-4568-a0ec-d58d51359df3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:40.519517 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.519495 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c46b50-6396-4568-a0ec-d58d51359df3-kube-api-access-8qwvx" (OuterVolumeSpecName: "kube-api-access-8qwvx") pod "58c46b50-6396-4568-a0ec-d58d51359df3" (UID: "58c46b50-6396-4568-a0ec-d58d51359df3"). InnerVolumeSpecName "kube-api-access-8qwvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:47:40.522556 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.522532 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-util" (OuterVolumeSpecName: "util") pod "58c46b50-6396-4568-a0ec-d58d51359df3" (UID: "58c46b50-6396-4568-a0ec-d58d51359df3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:47:40.619199 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.619134 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qwvx\" (UniqueName: \"kubernetes.io/projected/58c46b50-6396-4568-a0ec-d58d51359df3-kube-api-access-8qwvx\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:47:40.619199 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.619201 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-util\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:47:40.619439 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.619218 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58c46b50-6396-4568-a0ec-d58d51359df3-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:47:40.934166 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.934112 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" event={"ID":"382e3f4d-0714-4b70-8e27-d471c525de0e","Type":"ContainerStarted","Data":"ffb053ed170da50b15549b3ed61d77433822d15488e7036d9ca6a6a400f51bef"} Sep 16 13:47:40.936011 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.935981 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" event={"ID":"58c46b50-6396-4568-a0ec-d58d51359df3","Type":"ContainerDied","Data":"7ebfc98280007fa6323fc5c5bbf2b9911f953815a6483bdc63a74ac8bac052a6"} Sep 16 13:47:40.936135 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.936024 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xchs7" Sep 16 13:47:40.936135 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:40.936026 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ebfc98280007fa6323fc5c5bbf2b9911f953815a6483bdc63a74ac8bac052a6" Sep 16 13:47:41.942243 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:41.942135 2581 generic.go:356] "Generic (PLEG): container finished" podID="382e3f4d-0714-4b70-8e27-d471c525de0e" containerID="ffb053ed170da50b15549b3ed61d77433822d15488e7036d9ca6a6a400f51bef" exitCode=0 Sep 16 13:47:41.942597 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:41.942239 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" event={"ID":"382e3f4d-0714-4b70-8e27-d471c525de0e","Type":"ContainerDied","Data":"ffb053ed170da50b15549b3ed61d77433822d15488e7036d9ca6a6a400f51bef"} Sep 16 13:47:42.949581 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:42.949542 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" event={"ID":"382e3f4d-0714-4b70-8e27-d471c525de0e","Type":"ContainerStarted","Data":"903e1f0b1ff2a86da7f705805e1e9595fd79db9454cfa4da6e12ccd264d51e9c"} Sep 16 13:47:42.949946 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:42.949638 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:47:42.972018 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:42.971952 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" podStartSLOduration=2.5938774650000003 podStartE2EDuration="6.971935613s" totalImagesPullingTime="3.363988791s" totalInitContainerRuntime="1.014069357s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:42.969915788 +0000 UTC m=+296.933274615" watchObservedRunningTime="2026-09-16 13:47:42.971935613 +0000 UTC m=+296.935294524" Sep 16 13:47:46.461209 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:46.461183 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 13:47:46.465973 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:46.465944 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 13:47:46.471345 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:46.471321 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:47:46.474715 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:46.474696 2581 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 13:47:46.475517 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:46.475497 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:47:53.956833 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:47:53.956800 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-wm9r4" Sep 16 13:48:06.343180 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.343114 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t"] Sep 16 13:48:06.343645 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.343487 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c46b50-6396-4568-a0ec-d58d51359df3" containerName="pull" Sep 16 13:48:06.343645 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.343499 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c46b50-6396-4568-a0ec-d58d51359df3" containerName="pull" Sep 16 13:48:06.343645 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.343509 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c46b50-6396-4568-a0ec-d58d51359df3" containerName="util" Sep 16 13:48:06.343645 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.343516 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c46b50-6396-4568-a0ec-d58d51359df3" containerName="util" Sep 16 13:48:06.343645 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.343526 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c46b50-6396-4568-a0ec-d58d51359df3" containerName="extract" Sep 16 13:48:06.343645 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.343533 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c46b50-6396-4568-a0ec-d58d51359df3" containerName="extract" Sep 16 13:48:06.343645 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.343598 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c46b50-6396-4568-a0ec-d58d51359df3" containerName="extract" Sep 16 13:48:06.348426 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.348407 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:06.350879 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.350852 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 16 13:48:06.351005 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.350907 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 16 13:48:06.351005 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.350939 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-n25nr\"" Sep 16 13:48:06.357493 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.357468 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t"] Sep 16 13:48:06.419509 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.419470 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb85e53b-69b8-4562-8e76-d66b23610895-tls-certs\") pod \"kube-auth-proxy-66fc9bc5dc-c7k2t\" (UID: \"bb85e53b-69b8-4562-8e76-d66b23610895\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:06.419640 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.419525 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb85e53b-69b8-4562-8e76-d66b23610895-tmp\") pod \"kube-auth-proxy-66fc9bc5dc-c7k2t\" (UID: \"bb85e53b-69b8-4562-8e76-d66b23610895\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:06.419640 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.419586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnlrv\" (UniqueName: \"kubernetes.io/projected/bb85e53b-69b8-4562-8e76-d66b23610895-kube-api-access-pnlrv\") pod \"kube-auth-proxy-66fc9bc5dc-c7k2t\" (UID: \"bb85e53b-69b8-4562-8e76-d66b23610895\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:06.520123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.520097 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb85e53b-69b8-4562-8e76-d66b23610895-tmp\") pod \"kube-auth-proxy-66fc9bc5dc-c7k2t\" (UID: \"bb85e53b-69b8-4562-8e76-d66b23610895\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:06.520261 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.520171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnlrv\" (UniqueName: \"kubernetes.io/projected/bb85e53b-69b8-4562-8e76-d66b23610895-kube-api-access-pnlrv\") pod \"kube-auth-proxy-66fc9bc5dc-c7k2t\" (UID: \"bb85e53b-69b8-4562-8e76-d66b23610895\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:06.520261 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.520211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb85e53b-69b8-4562-8e76-d66b23610895-tls-certs\") pod \"kube-auth-proxy-66fc9bc5dc-c7k2t\" (UID: \"bb85e53b-69b8-4562-8e76-d66b23610895\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:06.520372 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:48:06.520315 2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 16 13:48:06.520372 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:48:06.520363 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb85e53b-69b8-4562-8e76-d66b23610895-tls-certs podName:bb85e53b-69b8-4562-8e76-d66b23610895 nodeName:}" failed. No retries permitted until 2026-09-16 13:48:07.020349007 +0000 UTC m=+320.983707817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/bb85e53b-69b8-4562-8e76-d66b23610895-tls-certs") pod "kube-auth-proxy-66fc9bc5dc-c7k2t" (UID: "bb85e53b-69b8-4562-8e76-d66b23610895") : secret "kube-auth-proxy-tls" not found Sep 16 13:48:06.522458 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.522440 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb85e53b-69b8-4562-8e76-d66b23610895-tmp\") pod \"kube-auth-proxy-66fc9bc5dc-c7k2t\" (UID: \"bb85e53b-69b8-4562-8e76-d66b23610895\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:06.530084 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:06.530064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnlrv\" (UniqueName: \"kubernetes.io/projected/bb85e53b-69b8-4562-8e76-d66b23610895-kube-api-access-pnlrv\") pod \"kube-auth-proxy-66fc9bc5dc-c7k2t\" (UID: \"bb85e53b-69b8-4562-8e76-d66b23610895\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:07.024584 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:07.024551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb85e53b-69b8-4562-8e76-d66b23610895-tls-certs\") pod \"kube-auth-proxy-66fc9bc5dc-c7k2t\" (UID: \"bb85e53b-69b8-4562-8e76-d66b23610895\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:07.027049 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:07.027023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb85e53b-69b8-4562-8e76-d66b23610895-tls-certs\") pod \"kube-auth-proxy-66fc9bc5dc-c7k2t\" (UID: \"bb85e53b-69b8-4562-8e76-d66b23610895\") " pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:07.258791 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:07.258752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" Sep 16 13:48:07.540349 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:07.540295 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t"] Sep 16 13:48:07.542740 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:48:07.542713 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb85e53b_69b8_4562_8e76_d66b23610895.slice/crio-3807ee1337995eba821cfa71ae3512deeb7e0083c9bd23e62aa0f822a942896e WatchSource:0}: Error finding container 3807ee1337995eba821cfa71ae3512deeb7e0083c9bd23e62aa0f822a942896e: Status 404 returned error can't find the container with id 3807ee1337995eba821cfa71ae3512deeb7e0083c9bd23e62aa0f822a942896e Sep 16 13:48:07.545017 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:07.544996 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:48:08.080509 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:08.080476 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" event={"ID":"bb85e53b-69b8-4562-8e76-d66b23610895","Type":"ContainerStarted","Data":"3807ee1337995eba821cfa71ae3512deeb7e0083c9bd23e62aa0f822a942896e"} Sep 16 13:48:11.096032 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:11.095989 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" event={"ID":"bb85e53b-69b8-4562-8e76-d66b23610895","Type":"ContainerStarted","Data":"b35800638bf1805888692f9a9748547e9b38fc3182018bb32b4f471915ece373"} Sep 16 13:48:11.113023 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:11.112974 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66fc9bc5dc-c7k2t" podStartSLOduration=2.14410794 podStartE2EDuration="5.112958578s" totalImagesPullingTime="2.968850638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:11.112167775 +0000 UTC m=+325.075526605" watchObservedRunningTime="2026-09-16 13:48:11.112958578 +0000 UTC m=+325.076317405" Sep 16 13:48:15.616235 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.616198 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf"] Sep 16 13:48:15.620275 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.620256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.623409 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.623385 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 16 13:48:15.624810 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.624788 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 16 13:48:15.624993 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.624972 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 13:48:15.625159 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.625120 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 16 13:48:15.625261 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.625240 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cwdfv\"" Sep 16 13:48:15.639730 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.639704 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf"] Sep 16 13:48:15.700753 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.700724 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dff9285c-d48d-437a-8a49-97faee1752d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.700877 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.700764 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dff9285c-d48d-437a-8a49-97faee1752d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.700877 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.700785 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dff9285c-d48d-437a-8a49-97faee1752d0-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.700950 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.700874 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4hgx\" (UniqueName: \"kubernetes.io/projected/dff9285c-d48d-437a-8a49-97faee1752d0-kube-api-access-t4hgx\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.700950 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.700904 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff9285c-d48d-437a-8a49-97faee1752d0-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.700950 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.700940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dff9285c-d48d-437a-8a49-97faee1752d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.701046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.700987 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff9285c-d48d-437a-8a49-97faee1752d0-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.801886 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.801859 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dff9285c-d48d-437a-8a49-97faee1752d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.801990 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.801909 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff9285c-d48d-437a-8a49-97faee1752d0-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.801990 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.801943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dff9285c-d48d-437a-8a49-97faee1752d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.801990 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.801968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dff9285c-d48d-437a-8a49-97faee1752d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.801990 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.801982 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dff9285c-d48d-437a-8a49-97faee1752d0-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.802205 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.801999 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4hgx\" (UniqueName: \"kubernetes.io/projected/dff9285c-d48d-437a-8a49-97faee1752d0-kube-api-access-t4hgx\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.802205 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.802018 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff9285c-d48d-437a-8a49-97faee1752d0-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.802725 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.802697 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dff9285c-d48d-437a-8a49-97faee1752d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.804444 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.804274 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff9285c-d48d-437a-8a49-97faee1752d0-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.804562 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.804486 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dff9285c-d48d-437a-8a49-97faee1752d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.804633 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.804580 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dff9285c-d48d-437a-8a49-97faee1752d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.804784 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.804767 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dff9285c-d48d-437a-8a49-97faee1752d0-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.822761 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.822727 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff9285c-d48d-437a-8a49-97faee1752d0-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.829538 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.829507 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4hgx\" (UniqueName: \"kubernetes.io/projected/dff9285c-d48d-437a-8a49-97faee1752d0-kube-api-access-t4hgx\") pod \"istiod-openshift-gateway-5677ccb49b-vxrxf\" (UID: \"dff9285c-d48d-437a-8a49-97faee1752d0\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:15.929832 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:15.929773 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:16.234012 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:16.233980 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf"] Sep 16 13:48:16.237930 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:48:16.237902 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff9285c_d48d_437a_8a49_97faee1752d0.slice/crio-cead4f801ad5ebe092f0a3933bcc02590b8cb6c8d2eb2cb1085984e05ab2cd9c WatchSource:0}: Error finding container cead4f801ad5ebe092f0a3933bcc02590b8cb6c8d2eb2cb1085984e05ab2cd9c: Status 404 returned error can't find the container with id cead4f801ad5ebe092f0a3933bcc02590b8cb6c8d2eb2cb1085984e05ab2cd9c Sep 16 13:48:17.128129 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:17.128090 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" event={"ID":"dff9285c-d48d-437a-8a49-97faee1752d0","Type":"ContainerStarted","Data":"cead4f801ad5ebe092f0a3933bcc02590b8cb6c8d2eb2cb1085984e05ab2cd9c"} Sep 16 13:48:18.438485 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:18.438450 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 13:48:18.438754 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:18.438520 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 13:48:19.141658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.141608 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" event={"ID":"dff9285c-d48d-437a-8a49-97faee1752d0","Type":"ContainerStarted","Data":"2501db687fe82a2cf4800d329a6a881bfb9ab5792c7adf030855a7dee51e24d7"} Sep 16 13:48:19.141860 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.141756 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:19.143701 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.143673 2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-vxrxf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:48:19.143821 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.143724 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" podUID="dff9285c-d48d-437a-8a49-97faee1752d0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:48:19.203132 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.203075 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" podStartSLOduration=2.004771183 podStartE2EDuration="4.203052813s" totalImagesPullingTime="2.19828163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:19.201774044 +0000 UTC m=+333.165132873" watchObservedRunningTime="2026-09-16 13:48:19.203052813 +0000 UTC m=+333.166411640" Sep 16 13:48:19.325354 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.325316 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77"] Sep 16 13:48:19.328736 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.328720 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:19.331289 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.331267 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 16 13:48:19.331523 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.331504 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-qn7wk\"" Sep 16 13:48:19.342086 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.342041 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77"] Sep 16 13:48:19.436383 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.436306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx62g\" (UniqueName: \"kubernetes.io/projected/4d68b82b-d7a5-44dc-a602-229ecfe490c7-kube-api-access-mx62g\") pod \"llmisvc-controller-manager-777c6bd7d5-4gx77\" (UID: \"4d68b82b-d7a5-44dc-a602-229ecfe490c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:19.436383 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.436354 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d68b82b-d7a5-44dc-a602-229ecfe490c7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4gx77\" (UID: \"4d68b82b-d7a5-44dc-a602-229ecfe490c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:19.542546 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.542504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx62g\" (UniqueName: \"kubernetes.io/projected/4d68b82b-d7a5-44dc-a602-229ecfe490c7-kube-api-access-mx62g\") pod \"llmisvc-controller-manager-777c6bd7d5-4gx77\" (UID: \"4d68b82b-d7a5-44dc-a602-229ecfe490c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:19.543004 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.542567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d68b82b-d7a5-44dc-a602-229ecfe490c7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4gx77\" (UID: \"4d68b82b-d7a5-44dc-a602-229ecfe490c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:19.543004 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:48:19.542760 2581 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 16 13:48:19.543004 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:48:19.542823 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d68b82b-d7a5-44dc-a602-229ecfe490c7-cert podName:4d68b82b-d7a5-44dc-a602-229ecfe490c7 nodeName:}" failed. No retries permitted until 2026-09-16 13:48:20.042801527 +0000 UTC m=+334.006160351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4d68b82b-d7a5-44dc-a602-229ecfe490c7-cert") pod "llmisvc-controller-manager-777c6bd7d5-4gx77" (UID: "4d68b82b-d7a5-44dc-a602-229ecfe490c7") : secret "llmisvc-webhook-server-cert" not found Sep 16 13:48:19.552742 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:19.552713 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx62g\" (UniqueName: \"kubernetes.io/projected/4d68b82b-d7a5-44dc-a602-229ecfe490c7-kube-api-access-mx62g\") pod \"llmisvc-controller-manager-777c6bd7d5-4gx77\" (UID: \"4d68b82b-d7a5-44dc-a602-229ecfe490c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:20.047585 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:20.047540 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d68b82b-d7a5-44dc-a602-229ecfe490c7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4gx77\" (UID: \"4d68b82b-d7a5-44dc-a602-229ecfe490c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:20.047758 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:48:20.047703 2581 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 16 13:48:20.047804 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:48:20.047770 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d68b82b-d7a5-44dc-a602-229ecfe490c7-cert podName:4d68b82b-d7a5-44dc-a602-229ecfe490c7 nodeName:}" failed. No retries permitted until 2026-09-16 13:48:21.047754723 +0000 UTC m=+335.011113533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4d68b82b-d7a5-44dc-a602-229ecfe490c7-cert") pod "llmisvc-controller-manager-777c6bd7d5-4gx77" (UID: "4d68b82b-d7a5-44dc-a602-229ecfe490c7") : secret "llmisvc-webhook-server-cert" not found Sep 16 13:48:20.146770 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:20.146740 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vxrxf" Sep 16 13:48:21.055200 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:21.055164 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d68b82b-d7a5-44dc-a602-229ecfe490c7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4gx77\" (UID: \"4d68b82b-d7a5-44dc-a602-229ecfe490c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:21.057707 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:21.057680 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d68b82b-d7a5-44dc-a602-229ecfe490c7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4gx77\" (UID: \"4d68b82b-d7a5-44dc-a602-229ecfe490c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:21.139509 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:21.139480 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:21.412210 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:21.412186 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77"] Sep 16 13:48:21.414370 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:48:21.414342 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4d68b82b_d7a5_44dc_a602_229ecfe490c7.slice/crio-3585328ea93245f0b1b3e2eb2a0e95516552a9082bcc8445c2284b563ef6cc87 WatchSource:0}: Error finding container 3585328ea93245f0b1b3e2eb2a0e95516552a9082bcc8445c2284b563ef6cc87: Status 404 returned error can't find the container with id 3585328ea93245f0b1b3e2eb2a0e95516552a9082bcc8445c2284b563ef6cc87 Sep 16 13:48:22.161482 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:22.161449 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" event={"ID":"4d68b82b-d7a5-44dc-a602-229ecfe490c7","Type":"ContainerStarted","Data":"3585328ea93245f0b1b3e2eb2a0e95516552a9082bcc8445c2284b563ef6cc87"} Sep 16 13:48:26.182555 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:26.182512 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" event={"ID":"4d68b82b-d7a5-44dc-a602-229ecfe490c7","Type":"ContainerStarted","Data":"4e12e0dde8630681d6245c10623073b2f353764832b2d58f856a315f86e57d10"} Sep 16 13:48:26.182986 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:26.182652 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:48:26.200132 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:26.200086 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" podStartSLOduration=3.338218859 podStartE2EDuration="7.20007141s" totalImagesPullingTime="3.861852551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:26.199675586 +0000 UTC m=+340.163034417" watchObservedRunningTime="2026-09-16 13:48:26.20007141 +0000 UTC m=+340.163430237" Sep 16 13:48:57.189557 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:48:57.189464 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4gx77" Sep 16 13:49:11.956725 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:11.956688 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr"] Sep 16 13:49:11.960446 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:11.960419 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:11.961910 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:11.961886 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr"] Sep 16 13:49:11.963064 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:11.963046 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:49:11.964103 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:11.964080 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:49:11.964220 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:11.964086 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h26rk\"" Sep 16 13:49:12.043587 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.043559 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr"] Sep 16 13:49:12.045040 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.045019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zgb\" (UniqueName: \"kubernetes.io/projected/dba84835-6edb-4601-a9c4-291292a54e11-kube-api-access-m7zgb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:12.045112 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.045090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:12.045112 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.045109 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:12.047004 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.046987 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.055187 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.055165 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr"] Sep 16 13:49:12.146266 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.146228 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.146394 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.146293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.146394 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.146331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zgb\" (UniqueName: \"kubernetes.io/projected/dba84835-6edb-4601-a9c4-291292a54e11-kube-api-access-m7zgb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:12.146394 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.146384 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xdp\" (UniqueName: \"kubernetes.io/projected/86336052-260d-4784-926a-5c95481ba620-kube-api-access-w2xdp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.146539 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.146461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:12.146539 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.146484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:12.146877 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.146856 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:12.146877 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.146872 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:12.148067 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.148047 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl"] Sep 16 13:49:12.151427 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.151412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.155504 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.155485 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zgb\" (UniqueName: \"kubernetes.io/projected/dba84835-6edb-4601-a9c4-291292a54e11-kube-api-access-m7zgb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:12.159798 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.159774 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl"] Sep 16 13:49:12.247073 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.247042 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.247227 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.247086 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.247227 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.247132 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.247227 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.247199 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xdp\" (UniqueName: \"kubernetes.io/projected/86336052-260d-4784-926a-5c95481ba620-kube-api-access-w2xdp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.247365 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.247238 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7g2j\" (UniqueName: \"kubernetes.io/projected/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-kube-api-access-t7g2j\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.247365 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.247357 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.247508 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.247485 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.247549 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.247498 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.248971 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.248952 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g"] Sep 16 13:49:12.252381 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.252366 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.258310 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.258289 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xdp\" (UniqueName: \"kubernetes.io/projected/86336052-260d-4784-926a-5c95481ba620-kube-api-access-w2xdp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.260181 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.260161 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g"] Sep 16 13:49:12.270500 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.270477 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:12.348730 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.348700 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.348889 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.348745 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.348889 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.348775 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7g2j\" (UniqueName: \"kubernetes.io/projected/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-kube-api-access-t7g2j\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.348889 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.348805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.348889 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.348840 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.349114 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.348941 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrvf\" (UniqueName: \"kubernetes.io/projected/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-kube-api-access-fdrvf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.349210 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.349175 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.349210 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.349190 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.355580 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.355560 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:12.358360 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.358336 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7g2j\" (UniqueName: \"kubernetes.io/projected/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-kube-api-access-t7g2j\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.450066 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.450036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrvf\" (UniqueName: \"kubernetes.io/projected/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-kube-api-access-fdrvf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.450247 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.450080 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.450247 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.450136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.450485 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.450466 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.450551 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.450497 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.459921 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.459897 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrvf\" (UniqueName: \"kubernetes.io/projected/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-kube-api-access-fdrvf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.467152 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.467123 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:12.545902 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.545870 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr"] Sep 16 13:49:12.547126 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:49:12.547084 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba84835_6edb_4601_a9c4_291292a54e11.slice/crio-9768719c66adc8829ef9352af1feb030a2541f14a53691a9c409c71cc44e778b WatchSource:0}: Error finding container 9768719c66adc8829ef9352af1feb030a2541f14a53691a9c409c71cc44e778b: Status 404 returned error can't find the container with id 9768719c66adc8829ef9352af1feb030a2541f14a53691a9c409c71cc44e778b Sep 16 13:49:12.568509 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.568488 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:12.628110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.627947 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr"] Sep 16 13:49:12.631458 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:49:12.631412 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86336052_260d_4784_926a_5c95481ba620.slice/crio-08bfa8c5e684bdefc735d2392c70b95d2e0394d049e2c8c03c110fe6004408f9 WatchSource:0}: Error finding container 08bfa8c5e684bdefc735d2392c70b95d2e0394d049e2c8c03c110fe6004408f9: Status 404 returned error can't find the container with id 08bfa8c5e684bdefc735d2392c70b95d2e0394d049e2c8c03c110fe6004408f9 Sep 16 13:49:12.738735 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.738708 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl"] Sep 16 13:49:12.740304 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:49:12.740276 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccb9a80_15f0_4a70_bee8_fbb1ef3643f7.slice/crio-e6b7e34a91d58dc1a15761391f29956dc5284c4454e5ded297457c4d2fb07bf4 WatchSource:0}: Error finding container e6b7e34a91d58dc1a15761391f29956dc5284c4454e5ded297457c4d2fb07bf4: Status 404 returned error can't find the container with id e6b7e34a91d58dc1a15761391f29956dc5284c4454e5ded297457c4d2fb07bf4 Sep 16 13:49:12.842067 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:12.842042 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g"] Sep 16 13:49:12.888849 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:49:12.888824 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c90ebc4_4b9d_48c7_932a_5d35fca34ee8.slice/crio-9174f2dc61974ebbb843e2c28d74518ab0c258f7d88ae279501bf08f58b7f283 WatchSource:0}: Error finding container 9174f2dc61974ebbb843e2c28d74518ab0c258f7d88ae279501bf08f58b7f283: Status 404 returned error can't find the container with id 9174f2dc61974ebbb843e2c28d74518ab0c258f7d88ae279501bf08f58b7f283 Sep 16 13:49:13.415028 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.414993 2581 generic.go:356] "Generic (PLEG): container finished" podID="8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" containerID="3cedc0c04852cd303f4d909c5db436d45b2fe4b0fc3bd7be8417fde165bc7d43" exitCode=0 Sep 16 13:49:13.415463 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.415078 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" event={"ID":"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7","Type":"ContainerDied","Data":"3cedc0c04852cd303f4d909c5db436d45b2fe4b0fc3bd7be8417fde165bc7d43"} Sep 16 13:49:13.415463 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.415126 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" event={"ID":"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7","Type":"ContainerStarted","Data":"e6b7e34a91d58dc1a15761391f29956dc5284c4454e5ded297457c4d2fb07bf4"} Sep 16 13:49:13.416762 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.416739 2581 generic.go:356] "Generic (PLEG): container finished" podID="86336052-260d-4784-926a-5c95481ba620" containerID="eef4e12713435bdc7559bd1fdb6cdb434c8970709eef01375a397b8cdd990f4a" exitCode=0 Sep 16 13:49:13.416859 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.416831 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" event={"ID":"86336052-260d-4784-926a-5c95481ba620","Type":"ContainerDied","Data":"eef4e12713435bdc7559bd1fdb6cdb434c8970709eef01375a397b8cdd990f4a"} Sep 16 13:49:13.416921 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.416883 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" event={"ID":"86336052-260d-4784-926a-5c95481ba620","Type":"ContainerStarted","Data":"08bfa8c5e684bdefc735d2392c70b95d2e0394d049e2c8c03c110fe6004408f9"} Sep 16 13:49:13.418456 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.418437 2581 generic.go:356] "Generic (PLEG): container finished" podID="5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" containerID="dea6b4be0178394181d36f77fb8257fac62782e39ad34db96b5e08b0bcb9db08" exitCode=0 Sep 16 13:49:13.418546 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.418517 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" event={"ID":"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8","Type":"ContainerDied","Data":"dea6b4be0178394181d36f77fb8257fac62782e39ad34db96b5e08b0bcb9db08"} Sep 16 13:49:13.418611 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.418555 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" event={"ID":"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8","Type":"ContainerStarted","Data":"9174f2dc61974ebbb843e2c28d74518ab0c258f7d88ae279501bf08f58b7f283"} Sep 16 13:49:13.420256 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.420236 2581 generic.go:356] "Generic (PLEG): container finished" podID="dba84835-6edb-4601-a9c4-291292a54e11" containerID="2f610b6373dfbfdff59b1f63ed50def63997641a1ee1f399510c9fc705594ce7" exitCode=0 Sep 16 13:49:13.420347 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.420330 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" event={"ID":"dba84835-6edb-4601-a9c4-291292a54e11","Type":"ContainerDied","Data":"2f610b6373dfbfdff59b1f63ed50def63997641a1ee1f399510c9fc705594ce7"} Sep 16 13:49:13.420390 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:13.420370 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" event={"ID":"dba84835-6edb-4601-a9c4-291292a54e11","Type":"ContainerStarted","Data":"9768719c66adc8829ef9352af1feb030a2541f14a53691a9c409c71cc44e778b"} Sep 16 13:49:14.427839 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:14.427807 2581 generic.go:356] "Generic (PLEG): container finished" podID="86336052-260d-4784-926a-5c95481ba620" containerID="db4df4d396b473722fc4c68b480c99be6e6a566f914481020e51cdc090097604" exitCode=0 Sep 16 13:49:14.428210 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:14.427891 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" event={"ID":"86336052-260d-4784-926a-5c95481ba620","Type":"ContainerDied","Data":"db4df4d396b473722fc4c68b480c99be6e6a566f914481020e51cdc090097604"} Sep 16 13:49:15.435034 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:15.435000 2581 generic.go:356] "Generic (PLEG): container finished" podID="8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" containerID="e52362157900378ec89f4dfc056c240f2385529a2713ab909fda9ef0749523f3" exitCode=0 Sep 16 13:49:15.435467 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:15.435077 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" event={"ID":"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7","Type":"ContainerDied","Data":"e52362157900378ec89f4dfc056c240f2385529a2713ab909fda9ef0749523f3"} Sep 16 13:49:15.437405 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:15.437381 2581 generic.go:356] "Generic (PLEG): container finished" podID="86336052-260d-4784-926a-5c95481ba620" containerID="ac6ae53613620245e25b7651691dd81b2ef1f21961c79ee7619c701d5e0b2a51" exitCode=0 Sep 16 13:49:15.437524 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:15.437463 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" event={"ID":"86336052-260d-4784-926a-5c95481ba620","Type":"ContainerDied","Data":"ac6ae53613620245e25b7651691dd81b2ef1f21961c79ee7619c701d5e0b2a51"} Sep 16 13:49:15.439204 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:15.439183 2581 generic.go:356] "Generic (PLEG): container finished" podID="5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" containerID="35d505fa1465c2ced1d677697db1667e1a923850625cdee04f324c82aa71ec3d" exitCode=0 Sep 16 13:49:15.439299 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:15.439241 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" event={"ID":"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8","Type":"ContainerDied","Data":"35d505fa1465c2ced1d677697db1667e1a923850625cdee04f324c82aa71ec3d"} Sep 16 13:49:16.446190 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.446133 2581 generic.go:356] "Generic (PLEG): container finished" podID="5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" containerID="84cb7da7af8b2b150f4c966b793e2b581c8e641e782364284f9e5024b61b4b02" exitCode=0 Sep 16 13:49:16.446608 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.446221 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" event={"ID":"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8","Type":"ContainerDied","Data":"84cb7da7af8b2b150f4c966b793e2b581c8e641e782364284f9e5024b61b4b02"} Sep 16 13:49:16.448305 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.448278 2581 generic.go:356] "Generic (PLEG): container finished" podID="8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" containerID="c351455a1a957309215487c05e448c3a63f04f168fafa26f26566a49429d94da" exitCode=0 Sep 16 13:49:16.448421 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.448360 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" event={"ID":"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7","Type":"ContainerDied","Data":"c351455a1a957309215487c05e448c3a63f04f168fafa26f26566a49429d94da"} Sep 16 13:49:16.577092 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.577068 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:16.689802 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.689771 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-util\") pod \"86336052-260d-4784-926a-5c95481ba620\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " Sep 16 13:49:16.689931 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.689899 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2xdp\" (UniqueName: \"kubernetes.io/projected/86336052-260d-4784-926a-5c95481ba620-kube-api-access-w2xdp\") pod \"86336052-260d-4784-926a-5c95481ba620\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " Sep 16 13:49:16.690001 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.689966 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-bundle\") pod \"86336052-260d-4784-926a-5c95481ba620\" (UID: \"86336052-260d-4784-926a-5c95481ba620\") " Sep 16 13:49:16.690490 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.690459 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-bundle" (OuterVolumeSpecName: "bundle") pod "86336052-260d-4784-926a-5c95481ba620" (UID: "86336052-260d-4784-926a-5c95481ba620"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:16.691974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.691945 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86336052-260d-4784-926a-5c95481ba620-kube-api-access-w2xdp" (OuterVolumeSpecName: "kube-api-access-w2xdp") pod "86336052-260d-4784-926a-5c95481ba620" (UID: "86336052-260d-4784-926a-5c95481ba620"). InnerVolumeSpecName "kube-api-access-w2xdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:16.695182 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.695132 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-util" (OuterVolumeSpecName: "util") pod "86336052-260d-4784-926a-5c95481ba620" (UID: "86336052-260d-4784-926a-5c95481ba620"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:16.790789 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.790762 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xdp\" (UniqueName: \"kubernetes.io/projected/86336052-260d-4784-926a-5c95481ba620-kube-api-access-w2xdp\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:16.790789 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.790788 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:16.790940 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:16.790798 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86336052-260d-4784-926a-5c95481ba620-util\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:17.459419 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.459317 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" event={"ID":"86336052-260d-4784-926a-5c95481ba620","Type":"ContainerDied","Data":"08bfa8c5e684bdefc735d2392c70b95d2e0394d049e2c8c03c110fe6004408f9"} Sep 16 13:49:17.459419 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.459379 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08bfa8c5e684bdefc735d2392c70b95d2e0394d049e2c8c03c110fe6004408f9" Sep 16 13:49:17.459926 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.459688 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7625wr" Sep 16 13:49:17.615034 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.615013 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:17.618674 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.618654 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:17.698843 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.698818 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-util\") pod \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " Sep 16 13:49:17.698970 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.698858 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdrvf\" (UniqueName: \"kubernetes.io/projected/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-kube-api-access-fdrvf\") pod \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " Sep 16 13:49:17.698970 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.698883 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7g2j\" (UniqueName: \"kubernetes.io/projected/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-kube-api-access-t7g2j\") pod \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " Sep 16 13:49:17.698970 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.698908 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-util\") pod \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " Sep 16 13:49:17.698970 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.698931 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-bundle\") pod \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\" (UID: \"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8\") " Sep 16 13:49:17.699203 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.698975 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-bundle\") pod \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\" (UID: \"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7\") " Sep 16 13:49:17.699539 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.699493 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-bundle" (OuterVolumeSpecName: "bundle") pod "5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" (UID: "5c90ebc4-4b9d-48c7-932a-5d35fca34ee8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:17.699799 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.699769 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-bundle" (OuterVolumeSpecName: "bundle") pod "8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" (UID: "8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:17.700953 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.700932 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-kube-api-access-t7g2j" (OuterVolumeSpecName: "kube-api-access-t7g2j") pod "8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" (UID: "8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7"). InnerVolumeSpecName "kube-api-access-t7g2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:17.701216 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.701195 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-kube-api-access-fdrvf" (OuterVolumeSpecName: "kube-api-access-fdrvf") pod "5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" (UID: "5c90ebc4-4b9d-48c7-932a-5d35fca34ee8"). InnerVolumeSpecName "kube-api-access-fdrvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:17.704809 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.704777 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-util" (OuterVolumeSpecName: "util") pod "8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" (UID: "8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:17.705031 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.705012 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-util" (OuterVolumeSpecName: "util") pod "5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" (UID: "5c90ebc4-4b9d-48c7-932a-5d35fca34ee8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:17.799866 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.799839 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:17.799866 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.799864 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-util\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:17.800025 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.799873 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrvf\" (UniqueName: \"kubernetes.io/projected/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-kube-api-access-fdrvf\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:17.800025 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.799882 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7g2j\" (UniqueName: \"kubernetes.io/projected/8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7-kube-api-access-t7g2j\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:17.800025 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.799890 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-util\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:17.800025 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:17.799900 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c90ebc4-4b9d-48c7-932a-5d35fca34ee8-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:18.466403 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:18.466367 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" event={"ID":"5c90ebc4-4b9d-48c7-932a-5d35fca34ee8","Type":"ContainerDied","Data":"9174f2dc61974ebbb843e2c28d74518ab0c258f7d88ae279501bf08f58b7f283"} Sep 16 13:49:18.466403 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:18.466410 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9174f2dc61974ebbb843e2c28d74518ab0c258f7d88ae279501bf08f58b7f283" Sep 16 13:49:18.466890 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:18.466378 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf8v9g" Sep 16 13:49:18.468319 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:18.468295 2581 generic.go:356] "Generic (PLEG): container finished" podID="dba84835-6edb-4601-a9c4-291292a54e11" containerID="a6006ff92fcb5dffa8b3a7366c313279b51a8f9e518931763ce1bd921812c886" exitCode=0 Sep 16 13:49:18.468445 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:18.468382 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" event={"ID":"dba84835-6edb-4601-a9c4-291292a54e11","Type":"ContainerDied","Data":"a6006ff92fcb5dffa8b3a7366c313279b51a8f9e518931763ce1bd921812c886"} Sep 16 13:49:18.470561 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:18.470539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" event={"ID":"8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7","Type":"ContainerDied","Data":"e6b7e34a91d58dc1a15761391f29956dc5284c4454e5ded297457c4d2fb07bf4"} Sep 16 13:49:18.470656 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:18.470580 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6b7e34a91d58dc1a15761391f29956dc5284c4454e5ded297457c4d2fb07bf4" Sep 16 13:49:18.470656 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:18.470547 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqltpl" Sep 16 13:49:19.479043 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:19.479006 2581 generic.go:356] "Generic (PLEG): container finished" podID="dba84835-6edb-4601-a9c4-291292a54e11" containerID="6a3d77c75345474dd4d531dd16c2f015ebc36db3ad483715678a3d2b0c14b034" exitCode=0 Sep 16 13:49:19.479462 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:19.479086 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" event={"ID":"dba84835-6edb-4601-a9c4-291292a54e11","Type":"ContainerDied","Data":"6a3d77c75345474dd4d531dd16c2f015ebc36db3ad483715678a3d2b0c14b034"} Sep 16 13:49:20.622037 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:20.622014 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:20.724063 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:20.724035 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-util\") pod \"dba84835-6edb-4601-a9c4-291292a54e11\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " Sep 16 13:49:20.724215 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:20.724092 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-bundle\") pod \"dba84835-6edb-4601-a9c4-291292a54e11\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " Sep 16 13:49:20.724215 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:20.724132 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7zgb\" (UniqueName: \"kubernetes.io/projected/dba84835-6edb-4601-a9c4-291292a54e11-kube-api-access-m7zgb\") pod \"dba84835-6edb-4601-a9c4-291292a54e11\" (UID: \"dba84835-6edb-4601-a9c4-291292a54e11\") " Sep 16 13:49:20.724531 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:20.724499 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-bundle" (OuterVolumeSpecName: "bundle") pod "dba84835-6edb-4601-a9c4-291292a54e11" (UID: "dba84835-6edb-4601-a9c4-291292a54e11"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:20.726239 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:20.726215 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba84835-6edb-4601-a9c4-291292a54e11-kube-api-access-m7zgb" (OuterVolumeSpecName: "kube-api-access-m7zgb") pod "dba84835-6edb-4601-a9c4-291292a54e11" (UID: "dba84835-6edb-4601-a9c4-291292a54e11"). InnerVolumeSpecName "kube-api-access-m7zgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:20.728950 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:20.728914 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-util" (OuterVolumeSpecName: "util") pod "dba84835-6edb-4601-a9c4-291292a54e11" (UID: "dba84835-6edb-4601-a9c4-291292a54e11"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:49:20.825349 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:20.825299 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-util\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:20.825349 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:20.825319 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dba84835-6edb-4601-a9c4-291292a54e11-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:20.825349 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:20.825329 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7zgb\" (UniqueName: \"kubernetes.io/projected/dba84835-6edb-4601-a9c4-291292a54e11-kube-api-access-m7zgb\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:21.491693 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:21.491657 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" event={"ID":"dba84835-6edb-4601-a9c4-291292a54e11","Type":"ContainerDied","Data":"9768719c66adc8829ef9352af1feb030a2541f14a53691a9c409c71cc44e778b"} Sep 16 13:49:21.491857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:21.491699 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n78nr" Sep 16 13:49:21.491857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:21.491708 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9768719c66adc8829ef9352af1feb030a2541f14a53691a9c409c71cc44e778b" Sep 16 13:49:29.993769 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.993734 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx"] Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994064 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86336052-260d-4784-926a-5c95481ba620" containerName="pull" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994076 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="86336052-260d-4784-926a-5c95481ba620" containerName="pull" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994083 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" containerName="extract" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994089 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" containerName="extract" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994098 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" containerName="util" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994104 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" containerName="util" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994114 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86336052-260d-4784-926a-5c95481ba620" containerName="extract" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994122 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="86336052-260d-4784-926a-5c95481ba620" containerName="extract" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994128 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba84835-6edb-4601-a9c4-291292a54e11" containerName="extract" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994133 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba84835-6edb-4601-a9c4-291292a54e11" containerName="extract" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994154 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86336052-260d-4784-926a-5c95481ba620" containerName="util" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994159 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="86336052-260d-4784-926a-5c95481ba620" containerName="util" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994164 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" containerName="pull" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994169 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" containerName="pull" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994181 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba84835-6edb-4601-a9c4-291292a54e11" containerName="util" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994189 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba84835-6edb-4601-a9c4-291292a54e11" containerName="util" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994198 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" containerName="pull" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994204 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" containerName="pull" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994213 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" containerName="util" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994218 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" containerName="util" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994225 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba84835-6edb-4601-a9c4-291292a54e11" containerName="pull" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994230 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba84835-6edb-4601-a9c4-291292a54e11" containerName="pull" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994237 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" containerName="extract" Sep 16 13:49:29.994232 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994242 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" containerName="extract" Sep 16 13:49:29.994939 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994298 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c90ebc4-4b9d-48c7-932a-5d35fca34ee8" containerName="extract" Sep 16 13:49:29.994939 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994307 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ccb9a80-15f0-4a70-bee8-fbb1ef3643f7" containerName="extract" Sep 16 13:49:29.994939 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994313 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba84835-6edb-4601-a9c4-291292a54e11" containerName="extract" Sep 16 13:49:29.994939 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.994322 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="86336052-260d-4784-926a-5c95481ba620" containerName="extract" Sep 16 13:49:29.997739 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:29.997721 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:30.000262 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:30.000240 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 13:49:30.001153 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:30.001114 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vhkdg\"" Sep 16 13:49:30.001238 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:30.001114 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 13:49:30.009172 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:30.009127 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx"] Sep 16 13:49:30.098329 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:30.098304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2w9\" (UniqueName: \"kubernetes.io/projected/a746ee87-8bca-4ab4-9ae7-b91a126f506c-kube-api-access-df2w9\") pod \"limitador-operator-controller-manager-74495c69f8-9vkdx\" (UID: \"a746ee87-8bca-4ab4-9ae7-b91a126f506c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:30.199594 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:30.199555 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2w9\" (UniqueName: \"kubernetes.io/projected/a746ee87-8bca-4ab4-9ae7-b91a126f506c-kube-api-access-df2w9\") pod \"limitador-operator-controller-manager-74495c69f8-9vkdx\" (UID: \"a746ee87-8bca-4ab4-9ae7-b91a126f506c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:30.211452 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:30.211427 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2w9\" (UniqueName: \"kubernetes.io/projected/a746ee87-8bca-4ab4-9ae7-b91a126f506c-kube-api-access-df2w9\") pod \"limitador-operator-controller-manager-74495c69f8-9vkdx\" (UID: \"a746ee87-8bca-4ab4-9ae7-b91a126f506c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:30.308173 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:30.308097 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:30.579042 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:30.579012 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx"] Sep 16 13:49:30.580407 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:49:30.580376 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda746ee87_8bca_4ab4_9ae7_b91a126f506c.slice/crio-0fc73200c27842df83e32cc5cd15cc54309a2bc151832364027ec9d997ffbfc1 WatchSource:0}: Error finding container 0fc73200c27842df83e32cc5cd15cc54309a2bc151832364027ec9d997ffbfc1: Status 404 returned error can't find the container with id 0fc73200c27842df83e32cc5cd15cc54309a2bc151832364027ec9d997ffbfc1 Sep 16 13:49:31.544757 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.544722 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" event={"ID":"a746ee87-8bca-4ab4-9ae7-b91a126f506c","Type":"ContainerStarted","Data":"0fc73200c27842df83e32cc5cd15cc54309a2bc151832364027ec9d997ffbfc1"} Sep 16 13:49:31.667700 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.667655 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77867fd8bc-2zfrg"] Sep 16 13:49:31.673873 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.673846 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.681472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.681408 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77867fd8bc-2zfrg"] Sep 16 13:49:31.810938 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.810754 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-oauth-serving-cert\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.810938 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.810812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggbkv\" (UniqueName: \"kubernetes.io/projected/1e422914-f6a8-4b84-b439-45c6e72a16df-kube-api-access-ggbkv\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.810938 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.810840 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e422914-f6a8-4b84-b439-45c6e72a16df-console-oauth-config\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.810938 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.810866 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-trusted-ca-bundle\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.810938 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.810927 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e422914-f6a8-4b84-b439-45c6e72a16df-console-serving-cert\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.811389 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.811029 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-service-ca\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.811389 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.811077 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-console-config\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.912536 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.912475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-oauth-serving-cert\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.912536 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.912537 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggbkv\" (UniqueName: \"kubernetes.io/projected/1e422914-f6a8-4b84-b439-45c6e72a16df-kube-api-access-ggbkv\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.912783 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.912564 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e422914-f6a8-4b84-b439-45c6e72a16df-console-oauth-config\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.912783 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.912586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-trusted-ca-bundle\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.912783 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.912618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e422914-f6a8-4b84-b439-45c6e72a16df-console-serving-cert\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.912783 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.912680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-service-ca\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.912783 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.912714 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-console-config\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.913560 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.913424 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-oauth-serving-cert\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.913560 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.913484 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-console-config\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.913763 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.913666 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-trusted-ca-bundle\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.913999 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.913950 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e422914-f6a8-4b84-b439-45c6e72a16df-service-ca\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.916662 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.916472 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e422914-f6a8-4b84-b439-45c6e72a16df-console-oauth-config\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.917041 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.917018 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e422914-f6a8-4b84-b439-45c6e72a16df-console-serving-cert\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.929774 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.929742 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggbkv\" (UniqueName: \"kubernetes.io/projected/1e422914-f6a8-4b84-b439-45c6e72a16df-kube-api-access-ggbkv\") pod \"console-77867fd8bc-2zfrg\" (UID: \"1e422914-f6a8-4b84-b439-45c6e72a16df\") " pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:31.989371 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:31.989335 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:32.280104 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:32.280016 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77867fd8bc-2zfrg"] Sep 16 13:49:32.282266 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:49:32.282242 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e422914_f6a8_4b84_b439_45c6e72a16df.slice/crio-a8b9a6a03eaec0b93d39c6ae37c0a474ffedcb316a1b1f9d49210d6bbd347c7d WatchSource:0}: Error finding container a8b9a6a03eaec0b93d39c6ae37c0a474ffedcb316a1b1f9d49210d6bbd347c7d: Status 404 returned error can't find the container with id a8b9a6a03eaec0b93d39c6ae37c0a474ffedcb316a1b1f9d49210d6bbd347c7d Sep 16 13:49:32.553738 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:32.553662 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77867fd8bc-2zfrg" event={"ID":"1e422914-f6a8-4b84-b439-45c6e72a16df","Type":"ContainerStarted","Data":"a8b9a6a03eaec0b93d39c6ae37c0a474ffedcb316a1b1f9d49210d6bbd347c7d"} Sep 16 13:49:33.560764 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:33.560674 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77867fd8bc-2zfrg" event={"ID":"1e422914-f6a8-4b84-b439-45c6e72a16df","Type":"ContainerStarted","Data":"e2ec4823403f036ab5ceb48cd4d102458cdd1e81f7059fcd26cf70d7e32c33fd"} Sep 16 13:49:33.562466 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:33.562440 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" event={"ID":"a746ee87-8bca-4ab4-9ae7-b91a126f506c","Type":"ContainerStarted","Data":"a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d"} Sep 16 13:49:33.562595 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:33.562577 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:33.583930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:33.583892 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77867fd8bc-2zfrg" podStartSLOduration=2.583878816 podStartE2EDuration="2.583878816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:33.582846519 +0000 UTC m=+407.546205346" watchObservedRunningTime="2026-09-16 13:49:33.583878816 +0000 UTC m=+407.547237637" Sep 16 13:49:33.606561 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:33.606512 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" podStartSLOduration=1.843294779 podStartE2EDuration="4.606498545s" totalImagesPullingTime="2.763203766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:33.60541394 +0000 UTC m=+407.568772769" watchObservedRunningTime="2026-09-16 13:49:33.606498545 +0000 UTC m=+407.569857373" Sep 16 13:49:41.989557 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:41.989523 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:41.989933 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:41.989619 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:41.994043 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:41.994020 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:42.616436 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:42.616405 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77867fd8bc-2zfrg" Sep 16 13:49:42.664216 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:42.664180 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f745c9f75-szlvf"] Sep 16 13:49:44.575859 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:44.575833 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:57.763378 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:57.763345 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct"] Sep 16 13:49:57.768131 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:57.768114 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:49:57.771110 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:57.771087 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-khthc\"" Sep 16 13:49:57.781333 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:57.781308 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct"] Sep 16 13:49:57.813794 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:57.813773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7kx\" (UniqueName: \"kubernetes.io/projected/ccd079e1-8b08-4d51-8061-b75b9349f093-kube-api-access-7l7kx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-z5dct\" (UID: \"ccd079e1-8b08-4d51-8061-b75b9349f093\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:49:57.813887 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:57.813812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccd079e1-8b08-4d51-8061-b75b9349f093-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-z5dct\" (UID: \"ccd079e1-8b08-4d51-8061-b75b9349f093\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:49:57.914385 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:57.914354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7kx\" (UniqueName: \"kubernetes.io/projected/ccd079e1-8b08-4d51-8061-b75b9349f093-kube-api-access-7l7kx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-z5dct\" (UID: \"ccd079e1-8b08-4d51-8061-b75b9349f093\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:49:57.914472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:57.914407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccd079e1-8b08-4d51-8061-b75b9349f093-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-z5dct\" (UID: \"ccd079e1-8b08-4d51-8061-b75b9349f093\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:49:57.914725 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:57.914709 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccd079e1-8b08-4d51-8061-b75b9349f093-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-z5dct\" (UID: \"ccd079e1-8b08-4d51-8061-b75b9349f093\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:49:57.923379 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:57.923354 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7kx\" (UniqueName: \"kubernetes.io/projected/ccd079e1-8b08-4d51-8061-b75b9349f093-kube-api-access-7l7kx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-z5dct\" (UID: \"ccd079e1-8b08-4d51-8061-b75b9349f093\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:49:58.077635 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.077567 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:49:58.360789 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.360762 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct"] Sep 16 13:49:58.361761 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:49:58.361735 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd079e1_8b08_4d51_8061_b75b9349f093.slice/crio-9f48f131ee7e0ba93838983502c37f947a585807437ee26b1b5fa441dbffd8d9 WatchSource:0}: Error finding container 9f48f131ee7e0ba93838983502c37f947a585807437ee26b1b5fa441dbffd8d9: Status 404 returned error can't find the container with id 9f48f131ee7e0ba93838983502c37f947a585807437ee26b1b5fa441dbffd8d9 Sep 16 13:49:58.703120 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.703035 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" event={"ID":"ccd079e1-8b08-4d51-8061-b75b9349f093","Type":"ContainerStarted","Data":"9f48f131ee7e0ba93838983502c37f947a585807437ee26b1b5fa441dbffd8d9"} Sep 16 13:49:58.860153 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.860112 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct"] Sep 16 13:49:58.868377 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.868350 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct"] Sep 16 13:49:58.883583 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.883558 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz"] Sep 16 13:49:58.888196 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.888175 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:49:58.912306 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.912271 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz"] Sep 16 13:49:58.915070 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.915041 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx"] Sep 16 13:49:58.915282 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.915260 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" podUID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" containerName="manager" containerID="cri-o://a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d" gracePeriod=2 Sep 16 13:49:58.921911 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.921885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqb5k\" (UniqueName: \"kubernetes.io/projected/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-kube-api-access-jqb5k\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kcpwz\" (UID: \"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:49:58.922190 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.921956 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kcpwz\" (UID: \"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:49:58.926575 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.926552 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx"] Sep 16 13:49:58.937356 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:49:58.937322 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9vkdx\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:58.946512 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.946491 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6"] Sep 16 13:49:58.947126 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.947104 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" containerName="manager" Sep 16 13:49:58.947126 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.947127 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" containerName="manager" Sep 16 13:49:58.947310 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.947259 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" containerName="manager" Sep 16 13:49:58.955086 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.955018 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6" Sep 16 13:49:58.958819 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:49:58.958778 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9vkdx\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:58.964829 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:58.964773 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6"] Sep 16 13:49:59.022583 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.022548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqb5k\" (UniqueName: \"kubernetes.io/projected/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-kube-api-access-jqb5k\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kcpwz\" (UID: \"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:49:59.022719 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.022625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kcpwz\" (UID: \"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:49:59.022719 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.022659 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4hk4\" (UniqueName: \"kubernetes.io/projected/b83b14bf-169c-4d36-8a6a-e3098c9c0841-kube-api-access-s4hk4\") pod \"limitador-operator-controller-manager-74495c69f8-nvld6\" (UID: \"b83b14bf-169c-4d36-8a6a-e3098c9c0841\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6" Sep 16 13:49:59.023065 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.023040 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kcpwz\" (UID: \"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:49:59.034231 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.034183 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqb5k\" (UniqueName: \"kubernetes.io/projected/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-kube-api-access-jqb5k\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kcpwz\" (UID: \"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:49:59.123817 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.123778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4hk4\" (UniqueName: \"kubernetes.io/projected/b83b14bf-169c-4d36-8a6a-e3098c9c0841-kube-api-access-s4hk4\") pod \"limitador-operator-controller-manager-74495c69f8-nvld6\" (UID: \"b83b14bf-169c-4d36-8a6a-e3098c9c0841\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6" Sep 16 13:49:59.133482 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.133455 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4hk4\" (UniqueName: \"kubernetes.io/projected/b83b14bf-169c-4d36-8a6a-e3098c9c0841-kube-api-access-s4hk4\") pod \"limitador-operator-controller-manager-74495c69f8-nvld6\" (UID: \"b83b14bf-169c-4d36-8a6a-e3098c9c0841\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6" Sep 16 13:49:59.164759 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.164732 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:59.167019 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:49:59.166983 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9vkdx\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:59.202185 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.202161 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:49:59.225023 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.224998 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df2w9\" (UniqueName: \"kubernetes.io/projected/a746ee87-8bca-4ab4-9ae7-b91a126f506c-kube-api-access-df2w9\") pod \"a746ee87-8bca-4ab4-9ae7-b91a126f506c\" (UID: \"a746ee87-8bca-4ab4-9ae7-b91a126f506c\") " Sep 16 13:49:59.227656 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.227628 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a746ee87-8bca-4ab4-9ae7-b91a126f506c-kube-api-access-df2w9" (OuterVolumeSpecName: "kube-api-access-df2w9") pod "a746ee87-8bca-4ab4-9ae7-b91a126f506c" (UID: "a746ee87-8bca-4ab4-9ae7-b91a126f506c"). InnerVolumeSpecName "kube-api-access-df2w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:59.289206 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.289175 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6" Sep 16 13:49:59.326026 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.325995 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df2w9\" (UniqueName: \"kubernetes.io/projected/a746ee87-8bca-4ab4-9ae7-b91a126f506c-kube-api-access-df2w9\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:49:59.497174 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.497128 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz"] Sep 16 13:49:59.498527 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:49:59.498499 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93718e3c_b21f_42e1_b5bf_fd8b0f81edb0.slice/crio-0f4424c4fe84b74dce7864aef78adf73a0f9dd3032dcae6c10bc27de4505f1b6 WatchSource:0}: Error finding container 0f4424c4fe84b74dce7864aef78adf73a0f9dd3032dcae6c10bc27de4505f1b6: Status 404 returned error can't find the container with id 0f4424c4fe84b74dce7864aef78adf73a0f9dd3032dcae6c10bc27de4505f1b6 Sep 16 13:49:59.580418 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.580357 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6"] Sep 16 13:49:59.581720 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:49:59.581693 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83b14bf_169c_4d36_8a6a_e3098c9c0841.slice/crio-c4f0deed4a244ca906e79b4716dd546f5820a42804357ed38f39f1156b40a51d WatchSource:0}: Error finding container c4f0deed4a244ca906e79b4716dd546f5820a42804357ed38f39f1156b40a51d: Status 404 returned error can't find the container with id c4f0deed4a244ca906e79b4716dd546f5820a42804357ed38f39f1156b40a51d Sep 16 13:49:59.711456 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.711414 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" event={"ID":"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0","Type":"ContainerStarted","Data":"0f4424c4fe84b74dce7864aef78adf73a0f9dd3032dcae6c10bc27de4505f1b6"} Sep 16 13:49:59.713500 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.713440 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6" event={"ID":"b83b14bf-169c-4d36-8a6a-e3098c9c0841","Type":"ContainerStarted","Data":"68235411d2e3b288b7385c7e1621f3fba5adbd88d5a4c08e1cbe33f139059c39"} Sep 16 13:49:59.713657 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.713560 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6" event={"ID":"b83b14bf-169c-4d36-8a6a-e3098c9c0841","Type":"ContainerStarted","Data":"c4f0deed4a244ca906e79b4716dd546f5820a42804357ed38f39f1156b40a51d"} Sep 16 13:49:59.713742 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.713655 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6" Sep 16 13:49:59.715013 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.714980 2581 generic.go:356] "Generic (PLEG): container finished" podID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" containerID="a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d" exitCode=0 Sep 16 13:49:59.715124 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.715036 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:59.715124 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.715046 2581 scope.go:117] "RemoveContainer" containerID="a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d" Sep 16 13:49:59.715730 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:49:59.715696 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9vkdx\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:49:59.725584 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.725567 2581 scope.go:117] "RemoveContainer" containerID="a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d" Sep 16 13:49:59.725898 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:49:59.725866 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d\": container with ID starting with a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d not found: ID does not exist" containerID="a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d" Sep 16 13:49:59.725995 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.725909 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d"} err="failed to get container status \"a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d\": rpc error: code = NotFound desc = could not find container \"a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d\": container with ID starting with a86b9d5b08e3e70b3ff79183d84d7d5d10c0bae1d2425a842ac09367ae51f30d not found: ID does not exist" Sep 16 13:49:59.736225 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:49:59.736180 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6" podStartSLOduration=1.736135934 podStartE2EDuration="1.736135934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:59.734818746 +0000 UTC m=+433.698177586" watchObservedRunningTime="2026-09-16 13:49:59.736135934 +0000 UTC m=+433.699494793" Sep 16 13:49:59.737034 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:49:59.737006 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9vkdx\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9vkdx" Sep 16 13:50:00.519642 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:00.519596 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a746ee87-8bca-4ab4-9ae7-b91a126f506c" path="/var/lib/kubelet/pods/a746ee87-8bca-4ab4-9ae7-b91a126f506c/volumes" Sep 16 13:50:02.738938 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:02.738903 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" event={"ID":"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0","Type":"ContainerStarted","Data":"f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253"} Sep 16 13:50:02.739446 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:02.738964 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:50:02.740751 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:02.740714 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" podUID="ccd079e1-8b08-4d51-8061-b75b9349f093" containerName="manager" containerID="cri-o://88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029" gracePeriod=2 Sep 16 13:50:02.766780 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:02.766723 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" podStartSLOduration=2.029606368 podStartE2EDuration="4.766705092s" totalImagesPullingTime="2.737098724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:02.764877851 +0000 UTC m=+436.728236677" watchObservedRunningTime="2026-09-16 13:50:02.766705092 +0000 UTC m=+436.730063919" Sep 16 13:50:02.976996 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:02.976967 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:50:02.979043 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:50:02.979005 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-z5dct\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="ccd079e1-8b08-4d51-8061-b75b9349f093" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:50:03.062656 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.062569 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l7kx\" (UniqueName: \"kubernetes.io/projected/ccd079e1-8b08-4d51-8061-b75b9349f093-kube-api-access-7l7kx\") pod \"ccd079e1-8b08-4d51-8061-b75b9349f093\" (UID: \"ccd079e1-8b08-4d51-8061-b75b9349f093\") " Sep 16 13:50:03.062656 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.062623 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccd079e1-8b08-4d51-8061-b75b9349f093-extensions-socket-volume\") pod \"ccd079e1-8b08-4d51-8061-b75b9349f093\" (UID: \"ccd079e1-8b08-4d51-8061-b75b9349f093\") " Sep 16 13:50:03.062930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.062906 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd079e1-8b08-4d51-8061-b75b9349f093-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ccd079e1-8b08-4d51-8061-b75b9349f093" (UID: "ccd079e1-8b08-4d51-8061-b75b9349f093"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:03.064637 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.064617 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd079e1-8b08-4d51-8061-b75b9349f093-kube-api-access-7l7kx" (OuterVolumeSpecName: "kube-api-access-7l7kx") pod "ccd079e1-8b08-4d51-8061-b75b9349f093" (UID: "ccd079e1-8b08-4d51-8061-b75b9349f093"). InnerVolumeSpecName "kube-api-access-7l7kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:03.163816 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.163781 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccd079e1-8b08-4d51-8061-b75b9349f093-extensions-socket-volume\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:03.163816 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.163810 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l7kx\" (UniqueName: \"kubernetes.io/projected/ccd079e1-8b08-4d51-8061-b75b9349f093-kube-api-access-7l7kx\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:03.748322 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.748286 2581 generic.go:356] "Generic (PLEG): container finished" podID="ccd079e1-8b08-4d51-8061-b75b9349f093" containerID="88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029" exitCode=2 Sep 16 13:50:03.748680 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.748341 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:50:03.748744 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.748630 2581 scope.go:117] "RemoveContainer" containerID="88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029" Sep 16 13:50:03.756131 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:50:03.752898 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-z5dct\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="ccd079e1-8b08-4d51-8061-b75b9349f093" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:50:03.763492 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:50:03.763463 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-z5dct\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="ccd079e1-8b08-4d51-8061-b75b9349f093" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-z5dct" Sep 16 13:50:03.765388 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.765369 2581 scope.go:117] "RemoveContainer" containerID="88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029" Sep 16 13:50:03.765619 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:50:03.765602 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029\": container with ID starting with 88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029 not found: ID does not exist" containerID="88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029" Sep 16 13:50:03.765670 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:03.765628 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029"} err="failed to get container status \"88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029\": rpc error: code = NotFound desc = could not find container \"88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029\": container with ID starting with 88012d3e40fde18c955ce7c748a4987cc1ebf20b48b3fa53a898b22f4875f029 not found: ID does not exist" Sep 16 13:50:04.515199 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:04.515167 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccd079e1-8b08-4d51-8061-b75b9349f093" path="/var/lib/kubelet/pods/ccd079e1-8b08-4d51-8061-b75b9349f093/volumes" Sep 16 13:50:07.687212 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:07.687170 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f745c9f75-szlvf" podUID="adca585f-b9eb-42b7-a1aa-430fd6940a58" containerName="console" containerID="cri-o://3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8" gracePeriod=15 Sep 16 13:50:07.937046 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:07.937024 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:50:08.006328 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.006302 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m842\" (UniqueName: \"kubernetes.io/projected/adca585f-b9eb-42b7-a1aa-430fd6940a58-kube-api-access-5m842\") pod \"adca585f-b9eb-42b7-a1aa-430fd6940a58\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " Sep 16 13:50:08.006497 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.006348 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-serving-cert\") pod \"adca585f-b9eb-42b7-a1aa-430fd6940a58\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " Sep 16 13:50:08.006497 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.006406 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-trusted-ca-bundle\") pod \"adca585f-b9eb-42b7-a1aa-430fd6940a58\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " Sep 16 13:50:08.006497 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.006428 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-oauth-config\") pod \"adca585f-b9eb-42b7-a1aa-430fd6940a58\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " Sep 16 13:50:08.006497 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.006449 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-config\") pod \"adca585f-b9eb-42b7-a1aa-430fd6940a58\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " Sep 16 13:50:08.006721 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.006504 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-oauth-serving-cert\") pod \"adca585f-b9eb-42b7-a1aa-430fd6940a58\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " Sep 16 13:50:08.006721 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.006527 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-service-ca\") pod \"adca585f-b9eb-42b7-a1aa-430fd6940a58\" (UID: \"adca585f-b9eb-42b7-a1aa-430fd6940a58\") " Sep 16 13:50:08.006966 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.006940 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "adca585f-b9eb-42b7-a1aa-430fd6940a58" (UID: "adca585f-b9eb-42b7-a1aa-430fd6940a58"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:50:08.006966 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.006888 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "adca585f-b9eb-42b7-a1aa-430fd6940a58" (UID: "adca585f-b9eb-42b7-a1aa-430fd6940a58"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:50:08.007100 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.006935 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-config" (OuterVolumeSpecName: "console-config") pod "adca585f-b9eb-42b7-a1aa-430fd6940a58" (UID: "adca585f-b9eb-42b7-a1aa-430fd6940a58"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:50:08.007100 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.007045 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-service-ca" (OuterVolumeSpecName: "service-ca") pod "adca585f-b9eb-42b7-a1aa-430fd6940a58" (UID: "adca585f-b9eb-42b7-a1aa-430fd6940a58"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:50:08.008564 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.008539 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "adca585f-b9eb-42b7-a1aa-430fd6940a58" (UID: "adca585f-b9eb-42b7-a1aa-430fd6940a58"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:50:08.008564 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.008550 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adca585f-b9eb-42b7-a1aa-430fd6940a58-kube-api-access-5m842" (OuterVolumeSpecName: "kube-api-access-5m842") pod "adca585f-b9eb-42b7-a1aa-430fd6940a58" (UID: "adca585f-b9eb-42b7-a1aa-430fd6940a58"). InnerVolumeSpecName "kube-api-access-5m842". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:08.008564 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.008557 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "adca585f-b9eb-42b7-a1aa-430fd6940a58" (UID: "adca585f-b9eb-42b7-a1aa-430fd6940a58"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:50:08.108064 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.108028 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-oauth-serving-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:08.108064 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.108062 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:08.108279 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.108076 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m842\" (UniqueName: \"kubernetes.io/projected/adca585f-b9eb-42b7-a1aa-430fd6940a58-kube-api-access-5m842\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:08.108279 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.108091 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-serving-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:08.108279 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.108106 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-trusted-ca-bundle\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:08.108279 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.108119 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-oauth-config\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:08.108279 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.108133 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adca585f-b9eb-42b7-a1aa-430fd6940a58-console-config\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:08.778651 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.778561 2581 generic.go:356] "Generic (PLEG): container finished" podID="adca585f-b9eb-42b7-a1aa-430fd6940a58" containerID="3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8" exitCode=0 Sep 16 13:50:08.778651 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.778616 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f745c9f75-szlvf" Sep 16 13:50:08.779059 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.778641 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f745c9f75-szlvf" event={"ID":"adca585f-b9eb-42b7-a1aa-430fd6940a58","Type":"ContainerDied","Data":"3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8"} Sep 16 13:50:08.779059 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.778695 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f745c9f75-szlvf" event={"ID":"adca585f-b9eb-42b7-a1aa-430fd6940a58","Type":"ContainerDied","Data":"d15e06d47c410f8e586f1d0054d415478b6e79db055484d1feadd79eb320b09d"} Sep 16 13:50:08.779059 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.778721 2581 scope.go:117] "RemoveContainer" containerID="3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8" Sep 16 13:50:08.787404 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.787387 2581 scope.go:117] "RemoveContainer" containerID="3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8" Sep 16 13:50:08.787681 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:50:08.787659 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8\": container with ID starting with 3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8 not found: ID does not exist" containerID="3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8" Sep 16 13:50:08.787767 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.787691 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8"} err="failed to get container status \"3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8\": rpc error: code = NotFound desc = could not find container \"3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8\": container with ID starting with 3a5c74d39f76b4c110c9f22a7ca2b26ad0156bdc4bdbdf9ca3588093d6ed48e8 not found: ID does not exist" Sep 16 13:50:08.800437 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.800413 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f745c9f75-szlvf"] Sep 16 13:50:08.808101 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:08.808081 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f745c9f75-szlvf"] Sep 16 13:50:10.515857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:10.515819 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adca585f-b9eb-42b7-a1aa-430fd6940a58" path="/var/lib/kubelet/pods/adca585f-b9eb-42b7-a1aa-430fd6940a58/volumes" Sep 16 13:50:10.726874 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:10.726843 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nvld6" Sep 16 13:50:13.753010 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:13.752982 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:50:26.799388 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:26.799346 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz"] Sep 16 13:50:26.799757 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:26.799593 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" podUID="93718e3c-b21f-42e1-b5bf-fd8b0f81edb0" containerName="manager" containerID="cri-o://f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253" gracePeriod=10 Sep 16 13:50:27.057057 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.057001 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:50:27.157691 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.157661 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqb5k\" (UniqueName: \"kubernetes.io/projected/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-kube-api-access-jqb5k\") pod \"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0\" (UID: \"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0\") " Sep 16 13:50:27.157821 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.157744 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-extensions-socket-volume\") pod \"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0\" (UID: \"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0\") " Sep 16 13:50:27.158227 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.158195 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "93718e3c-b21f-42e1-b5bf-fd8b0f81edb0" (UID: "93718e3c-b21f-42e1-b5bf-fd8b0f81edb0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:27.159656 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.159630 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-kube-api-access-jqb5k" (OuterVolumeSpecName: "kube-api-access-jqb5k") pod "93718e3c-b21f-42e1-b5bf-fd8b0f81edb0" (UID: "93718e3c-b21f-42e1-b5bf-fd8b0f81edb0"). InnerVolumeSpecName "kube-api-access-jqb5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:27.258833 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.258805 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqb5k\" (UniqueName: \"kubernetes.io/projected/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-kube-api-access-jqb5k\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:27.258833 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.258829 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0-extensions-socket-volume\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:50:27.883086 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.883051 2581 generic.go:356] "Generic (PLEG): container finished" podID="93718e3c-b21f-42e1-b5bf-fd8b0f81edb0" containerID="f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253" exitCode=0 Sep 16 13:50:27.883577 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.883100 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" Sep 16 13:50:27.883577 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.883100 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" event={"ID":"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0","Type":"ContainerDied","Data":"f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253"} Sep 16 13:50:27.883577 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.883199 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz" event={"ID":"93718e3c-b21f-42e1-b5bf-fd8b0f81edb0","Type":"ContainerDied","Data":"0f4424c4fe84b74dce7864aef78adf73a0f9dd3032dcae6c10bc27de4505f1b6"} Sep 16 13:50:27.883577 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.883224 2581 scope.go:117] "RemoveContainer" containerID="f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253" Sep 16 13:50:27.892246 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.892229 2581 scope.go:117] "RemoveContainer" containerID="f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253" Sep 16 13:50:27.892496 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:50:27.892477 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253\": container with ID starting with f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253 not found: ID does not exist" containerID="f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253" Sep 16 13:50:27.892552 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.892508 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253"} err="failed to get container status \"f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253\": rpc error: code = NotFound desc = could not find container \"f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253\": container with ID starting with f186d9bca2c9d41ba9387351ecf30a44bdfa059e1a14dc763d0adce046b05253 not found: ID does not exist" Sep 16 13:50:27.915646 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.915622 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz"] Sep 16 13:50:27.919988 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:27.919964 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kcpwz"] Sep 16 13:50:28.515597 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:28.515564 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93718e3c-b21f-42e1-b5bf-fd8b0f81edb0" path="/var/lib/kubelet/pods/93718e3c-b21f-42e1-b5bf-fd8b0f81edb0/volumes" Sep 16 13:50:43.959356 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.959327 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lnhzf"] Sep 16 13:50:43.959922 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.959801 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adca585f-b9eb-42b7-a1aa-430fd6940a58" containerName="console" Sep 16 13:50:43.959922 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.959819 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="adca585f-b9eb-42b7-a1aa-430fd6940a58" containerName="console" Sep 16 13:50:43.959922 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.959834 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccd079e1-8b08-4d51-8061-b75b9349f093" containerName="manager" Sep 16 13:50:43.959922 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.959844 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccd079e1-8b08-4d51-8061-b75b9349f093" containerName="manager" Sep 16 13:50:43.959922 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.959864 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93718e3c-b21f-42e1-b5bf-fd8b0f81edb0" containerName="manager" Sep 16 13:50:43.959922 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.959873 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="93718e3c-b21f-42e1-b5bf-fd8b0f81edb0" containerName="manager" Sep 16 13:50:43.960263 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.959966 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccd079e1-8b08-4d51-8061-b75b9349f093" containerName="manager" Sep 16 13:50:43.960263 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.959980 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="adca585f-b9eb-42b7-a1aa-430fd6940a58" containerName="console" Sep 16 13:50:43.960263 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.959994 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="93718e3c-b21f-42e1-b5bf-fd8b0f81edb0" containerName="manager" Sep 16 13:50:43.963054 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.963035 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:50:43.965589 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.965567 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vfw4f\"" Sep 16 13:50:43.965687 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.965571 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 16 13:50:43.978621 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:43.978598 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lnhzf"] Sep 16 13:50:44.063199 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:44.063170 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lnhzf"] Sep 16 13:50:44.090490 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:44.090461 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rbv\" (UniqueName: \"kubernetes.io/projected/6d93fdef-4096-463e-b671-a5fa6acad0b2-kube-api-access-d8rbv\") pod \"limitador-limitador-8588749977-lnhzf\" (UID: \"6d93fdef-4096-463e-b671-a5fa6acad0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:50:44.090490 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:44.090491 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d93fdef-4096-463e-b671-a5fa6acad0b2-config-file\") pod \"limitador-limitador-8588749977-lnhzf\" (UID: \"6d93fdef-4096-463e-b671-a5fa6acad0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:50:44.190976 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:44.190945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rbv\" (UniqueName: \"kubernetes.io/projected/6d93fdef-4096-463e-b671-a5fa6acad0b2-kube-api-access-d8rbv\") pod \"limitador-limitador-8588749977-lnhzf\" (UID: \"6d93fdef-4096-463e-b671-a5fa6acad0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:50:44.191130 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:44.190981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d93fdef-4096-463e-b671-a5fa6acad0b2-config-file\") pod \"limitador-limitador-8588749977-lnhzf\" (UID: \"6d93fdef-4096-463e-b671-a5fa6acad0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:50:44.191662 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:44.191638 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d93fdef-4096-463e-b671-a5fa6acad0b2-config-file\") pod \"limitador-limitador-8588749977-lnhzf\" (UID: \"6d93fdef-4096-463e-b671-a5fa6acad0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:50:44.199704 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:44.199677 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rbv\" (UniqueName: \"kubernetes.io/projected/6d93fdef-4096-463e-b671-a5fa6acad0b2-kube-api-access-d8rbv\") pod \"limitador-limitador-8588749977-lnhzf\" (UID: \"6d93fdef-4096-463e-b671-a5fa6acad0b2\") " pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:50:44.273575 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:44.273518 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:50:44.553466 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:44.553443 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lnhzf"] Sep 16 13:50:44.555642 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:50:44.555613 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d93fdef_4096_463e_b671_a5fa6acad0b2.slice/crio-a03edf5ee9ae9d452175e46f4f7e8f81489c7c1f1cf4e1cfdb2752ca9b9e7615 WatchSource:0}: Error finding container a03edf5ee9ae9d452175e46f4f7e8f81489c7c1f1cf4e1cfdb2752ca9b9e7615: Status 404 returned error can't find the container with id a03edf5ee9ae9d452175e46f4f7e8f81489c7c1f1cf4e1cfdb2752ca9b9e7615 Sep 16 13:50:44.975654 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:44.975618 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" event={"ID":"6d93fdef-4096-463e-b671-a5fa6acad0b2","Type":"ContainerStarted","Data":"a03edf5ee9ae9d452175e46f4f7e8f81489c7c1f1cf4e1cfdb2752ca9b9e7615"} Sep 16 13:50:46.987643 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:46.987610 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" event={"ID":"6d93fdef-4096-463e-b671-a5fa6acad0b2","Type":"ContainerStarted","Data":"854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c"} Sep 16 13:50:46.987981 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:46.987739 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:50:47.010486 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:47.010440 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" podStartSLOduration=2.174220391 podStartE2EDuration="4.010421739s" totalImagesPullingTime="1.836201348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:47.00896128 +0000 UTC m=+480.972320148" watchObservedRunningTime="2026-09-16 13:50:47.010421739 +0000 UTC m=+480.973780565" Sep 16 13:50:57.994529 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:57.994496 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:50:59.667101 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:59.667062 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lnhzf"] Sep 16 13:50:59.667502 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:50:59.667269 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" podUID="6d93fdef-4096-463e-b671-a5fa6acad0b2" containerName="limitador" containerID="cri-o://854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c" gracePeriod=30 Sep 16 13:51:00.213381 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:00.213357 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:51:00.217206 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:00.217184 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d93fdef-4096-463e-b671-a5fa6acad0b2-config-file\") pod \"6d93fdef-4096-463e-b671-a5fa6acad0b2\" (UID: \"6d93fdef-4096-463e-b671-a5fa6acad0b2\") " Sep 16 13:51:00.217306 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:00.217289 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8rbv\" (UniqueName: \"kubernetes.io/projected/6d93fdef-4096-463e-b671-a5fa6acad0b2-kube-api-access-d8rbv\") pod \"6d93fdef-4096-463e-b671-a5fa6acad0b2\" (UID: \"6d93fdef-4096-463e-b671-a5fa6acad0b2\") " Sep 16 13:51:00.217526 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:00.217503 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d93fdef-4096-463e-b671-a5fa6acad0b2-config-file" (OuterVolumeSpecName: "config-file") pod "6d93fdef-4096-463e-b671-a5fa6acad0b2" (UID: "6d93fdef-4096-463e-b671-a5fa6acad0b2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:51:00.219115 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:00.219094 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d93fdef-4096-463e-b671-a5fa6acad0b2-kube-api-access-d8rbv" (OuterVolumeSpecName: "kube-api-access-d8rbv") pod "6d93fdef-4096-463e-b671-a5fa6acad0b2" (UID: "6d93fdef-4096-463e-b671-a5fa6acad0b2"). InnerVolumeSpecName "kube-api-access-d8rbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:51:00.318488 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:00.318394 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8rbv\" (UniqueName: \"kubernetes.io/projected/6d93fdef-4096-463e-b671-a5fa6acad0b2-kube-api-access-d8rbv\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:51:00.318488 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:00.318427 2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6d93fdef-4096-463e-b671-a5fa6acad0b2-config-file\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:51:01.064060 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:01.064024 2581 generic.go:356] "Generic (PLEG): container finished" podID="6d93fdef-4096-463e-b671-a5fa6acad0b2" containerID="854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c" exitCode=0 Sep 16 13:51:01.064493 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:01.064067 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" Sep 16 13:51:01.064493 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:01.064115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" event={"ID":"6d93fdef-4096-463e-b671-a5fa6acad0b2","Type":"ContainerDied","Data":"854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c"} Sep 16 13:51:01.064493 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:01.064180 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lnhzf" event={"ID":"6d93fdef-4096-463e-b671-a5fa6acad0b2","Type":"ContainerDied","Data":"a03edf5ee9ae9d452175e46f4f7e8f81489c7c1f1cf4e1cfdb2752ca9b9e7615"} Sep 16 13:51:01.064493 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:01.064203 2581 scope.go:117] "RemoveContainer" containerID="854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c" Sep 16 13:51:01.073071 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:01.073053 2581 scope.go:117] "RemoveContainer" containerID="854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c" Sep 16 13:51:01.073357 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:51:01.073337 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c\": container with ID starting with 854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c not found: ID does not exist" containerID="854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c" Sep 16 13:51:01.073425 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:01.073364 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c"} err="failed to get container status \"854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c\": rpc error: code = NotFound desc = could not find container \"854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c\": container with ID starting with 854a20c1607b60f22cc98f2d8919293418fee72e42bf05942dcb3547e223570c not found: ID does not exist" Sep 16 13:51:01.083478 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:01.083455 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lnhzf"] Sep 16 13:51:01.088871 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:01.088850 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lnhzf"] Sep 16 13:51:02.515467 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:02.515436 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d93fdef-4096-463e-b671-a5fa6acad0b2" path="/var/lib/kubelet/pods/6d93fdef-4096-463e-b671-a5fa6acad0b2/volumes" Sep 16 13:51:04.840344 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.840311 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd"] Sep 16 13:51:04.840717 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.840665 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d93fdef-4096-463e-b671-a5fa6acad0b2" containerName="limitador" Sep 16 13:51:04.840717 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.840676 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d93fdef-4096-463e-b671-a5fa6acad0b2" containerName="limitador" Sep 16 13:51:04.840799 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.840747 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d93fdef-4096-463e-b671-a5fa6acad0b2" containerName="limitador" Sep 16 13:51:04.845335 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.845314 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" Sep 16 13:51:04.849901 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.849878 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 16 13:51:04.850105 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.850085 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-hzxtt\"" Sep 16 13:51:04.850653 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.850634 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:51:04.856853 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.856831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n6nv\" (UniqueName: \"kubernetes.io/projected/5ed7672c-8453-4190-9f24-19f007282c60-kube-api-access-9n6nv\") pod \"postgres-69fb4594cf-2pwzd\" (UID: \"5ed7672c-8453-4190-9f24-19f007282c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" Sep 16 13:51:04.856941 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.856859 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5ed7672c-8453-4190-9f24-19f007282c60-data\") pod \"postgres-69fb4594cf-2pwzd\" (UID: \"5ed7672c-8453-4190-9f24-19f007282c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" Sep 16 13:51:04.857686 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.857670 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:51:04.862291 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.862271 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd"] Sep 16 13:51:04.957381 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.957354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n6nv\" (UniqueName: \"kubernetes.io/projected/5ed7672c-8453-4190-9f24-19f007282c60-kube-api-access-9n6nv\") pod \"postgres-69fb4594cf-2pwzd\" (UID: \"5ed7672c-8453-4190-9f24-19f007282c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" Sep 16 13:51:04.957527 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.957386 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5ed7672c-8453-4190-9f24-19f007282c60-data\") pod \"postgres-69fb4594cf-2pwzd\" (UID: \"5ed7672c-8453-4190-9f24-19f007282c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" Sep 16 13:51:04.957807 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.957791 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5ed7672c-8453-4190-9f24-19f007282c60-data\") pod \"postgres-69fb4594cf-2pwzd\" (UID: \"5ed7672c-8453-4190-9f24-19f007282c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" Sep 16 13:51:04.968981 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:04.968950 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n6nv\" (UniqueName: \"kubernetes.io/projected/5ed7672c-8453-4190-9f24-19f007282c60-kube-api-access-9n6nv\") pod \"postgres-69fb4594cf-2pwzd\" (UID: \"5ed7672c-8453-4190-9f24-19f007282c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" Sep 16 13:51:05.155050 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:05.154963 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" Sep 16 13:51:05.425834 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:05.425806 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd"] Sep 16 13:51:05.427725 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:51:05.427701 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed7672c_8453_4190_9f24_19f007282c60.slice/crio-e40c317c072e806014e96014ed40ecd824426afb20aa9bd40664fe9539574250 WatchSource:0}: Error finding container e40c317c072e806014e96014ed40ecd824426afb20aa9bd40664fe9539574250: Status 404 returned error can't find the container with id e40c317c072e806014e96014ed40ecd824426afb20aa9bd40664fe9539574250 Sep 16 13:51:06.092370 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:06.092331 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" event={"ID":"5ed7672c-8453-4190-9f24-19f007282c60","Type":"ContainerStarted","Data":"e40c317c072e806014e96014ed40ecd824426afb20aa9bd40664fe9539574250"} Sep 16 13:51:11.124422 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:11.124383 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" event={"ID":"5ed7672c-8453-4190-9f24-19f007282c60","Type":"ContainerStarted","Data":"1a6a567b2e00fcb9c478ce3a306de2dd50d90fb4e1aa1e645c4de12082efe303"} Sep 16 13:51:11.124900 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:11.124481 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" Sep 16 13:51:11.142104 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:11.142062 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" podStartSLOduration=2.381201192 podStartE2EDuration="7.142049436s" totalImagesPullingTime="4.760848244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:11.139665276 +0000 UTC m=+505.103024106" watchObservedRunningTime="2026-09-16 13:51:11.142049436 +0000 UTC m=+505.105408264" Sep 16 13:51:17.158027 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:17.157998 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2pwzd" Sep 16 13:51:24.864336 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:24.864302 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5664c89cd7-ltml9"] Sep 16 13:51:24.886597 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:24.886572 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5664c89cd7-ltml9"] Sep 16 13:51:24.886746 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:24.886661 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:24.889542 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:24.889519 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 13:51:24.890734 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:24.890711 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 13:51:24.890825 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:24.890788 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 13:51:24.890889 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:24.890868 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-brhwx\"" Sep 16 13:51:24.924223 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:24.924202 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7gv\" (UniqueName: \"kubernetes.io/projected/1596e27c-9005-46a6-937d-9dab88cb2c6a-kube-api-access-6z7gv\") pod \"maas-controller-5664c89cd7-ltml9\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:24.924329 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:24.924255 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-webhook-cert\") pod \"maas-controller-5664c89cd7-ltml9\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:24.924329 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:24.924301 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-metrics-certs\") pod \"maas-controller-5664c89cd7-ltml9\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:25.013675 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.013644 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d96bd79db-k7vp7"] Sep 16 13:51:25.017200 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.017183 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.025009 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.024976 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7gv\" (UniqueName: \"kubernetes.io/projected/1596e27c-9005-46a6-937d-9dab88cb2c6a-kube-api-access-6z7gv\") pod \"maas-controller-5664c89cd7-ltml9\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:25.025170 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.025061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-webhook-cert\") pod \"maas-controller-5664c89cd7-ltml9\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:25.025170 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.025107 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-metrics-certs\") pod \"maas-controller-5664c89cd7-ltml9\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:25.027872 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.027854 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-metrics-certs\") pod \"maas-controller-5664c89cd7-ltml9\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:25.027955 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.027908 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-webhook-cert\") pod \"maas-controller-5664c89cd7-ltml9\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:25.028612 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.028592 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-k7vp7"] Sep 16 13:51:25.034425 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.034401 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7gv\" (UniqueName: \"kubernetes.io/projected/1596e27c-9005-46a6-937d-9dab88cb2c6a-kube-api-access-6z7gv\") pod \"maas-controller-5664c89cd7-ltml9\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:25.126274 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.126218 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-metrics-certs\") pod \"maas-controller-7d96bd79db-k7vp7\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.126374 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.126275 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-webhook-cert\") pod \"maas-controller-7d96bd79db-k7vp7\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.126374 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.126293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqnnl\" (UniqueName: \"kubernetes.io/projected/b99bc9d3-c7cd-4e4d-84b5-516314915be2-kube-api-access-fqnnl\") pod \"maas-controller-7d96bd79db-k7vp7\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.196224 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.196192 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:25.227450 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.227425 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-webhook-cert\") pod \"maas-controller-7d96bd79db-k7vp7\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.227544 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.227460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqnnl\" (UniqueName: \"kubernetes.io/projected/b99bc9d3-c7cd-4e4d-84b5-516314915be2-kube-api-access-fqnnl\") pod \"maas-controller-7d96bd79db-k7vp7\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.227544 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.227525 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-metrics-certs\") pod \"maas-controller-7d96bd79db-k7vp7\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.229930 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.229901 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-webhook-cert\") pod \"maas-controller-7d96bd79db-k7vp7\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.230054 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.229938 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-metrics-certs\") pod \"maas-controller-7d96bd79db-k7vp7\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.237868 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.237847 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqnnl\" (UniqueName: \"kubernetes.io/projected/b99bc9d3-c7cd-4e4d-84b5-516314915be2-kube-api-access-fqnnl\") pod \"maas-controller-7d96bd79db-k7vp7\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.326693 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.326667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:25.467249 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.467222 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5664c89cd7-ltml9"] Sep 16 13:51:25.468761 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:51:25.468734 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1596e27c_9005_46a6_937d_9dab88cb2c6a.slice/crio-43eb85777ed6973263bb650976c8d818411acca97767a9ed7fbb4a66587f1c0f WatchSource:0}: Error finding container 43eb85777ed6973263bb650976c8d818411acca97767a9ed7fbb4a66587f1c0f: Status 404 returned error can't find the container with id 43eb85777ed6973263bb650976c8d818411acca97767a9ed7fbb4a66587f1c0f Sep 16 13:51:25.599587 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:25.599565 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-k7vp7"] Sep 16 13:51:25.600407 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:51:25.600387 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99bc9d3_c7cd_4e4d_84b5_516314915be2.slice/crio-97f53c5fa7ae4d15af1c646f0ae33c073a6af07de885be4628337ec0722397f3 WatchSource:0}: Error finding container 97f53c5fa7ae4d15af1c646f0ae33c073a6af07de885be4628337ec0722397f3: Status 404 returned error can't find the container with id 97f53c5fa7ae4d15af1c646f0ae33c073a6af07de885be4628337ec0722397f3 Sep 16 13:51:26.214035 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:26.213960 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5664c89cd7-ltml9" event={"ID":"1596e27c-9005-46a6-937d-9dab88cb2c6a","Type":"ContainerStarted","Data":"43eb85777ed6973263bb650976c8d818411acca97767a9ed7fbb4a66587f1c0f"} Sep 16 13:51:26.216748 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:26.216688 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" event={"ID":"b99bc9d3-c7cd-4e4d-84b5-516314915be2","Type":"ContainerStarted","Data":"97f53c5fa7ae4d15af1c646f0ae33c073a6af07de885be4628337ec0722397f3"} Sep 16 13:51:28.229455 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:28.229406 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5664c89cd7-ltml9" event={"ID":"1596e27c-9005-46a6-937d-9dab88cb2c6a","Type":"ContainerStarted","Data":"d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21"} Sep 16 13:51:28.229927 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:28.229543 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:28.231909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:28.231881 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" event={"ID":"b99bc9d3-c7cd-4e4d-84b5-516314915be2","Type":"ContainerStarted","Data":"73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0"} Sep 16 13:51:28.232018 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:28.231935 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:28.256693 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:28.256658 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5664c89cd7-ltml9" podStartSLOduration=1.61848663 podStartE2EDuration="4.256646791s" totalImagesPullingTime="2.638160161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:28.255842877 +0000 UTC m=+522.219201718" watchObservedRunningTime="2026-09-16 13:51:28.256646791 +0000 UTC m=+522.220005617" Sep 16 13:51:28.275855 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:28.275800 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" podStartSLOduration=1.762335682 podStartE2EDuration="4.275786076s" totalImagesPullingTime="2.513450394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:28.274507326 +0000 UTC m=+522.237866179" watchObservedRunningTime="2026-09-16 13:51:28.275786076 +0000 UTC m=+522.239144902" Sep 16 13:51:39.240242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.240168 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:39.240242 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.240243 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 13:51:39.332429 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.332392 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5664c89cd7-ltml9"] Sep 16 13:51:39.332681 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.332652 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5664c89cd7-ltml9" podUID="1596e27c-9005-46a6-937d-9dab88cb2c6a" containerName="manager" containerID="cri-o://d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21" gracePeriod=30 Sep 16 13:51:39.575470 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.575444 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:39.661247 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.661217 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-metrics-certs\") pod \"1596e27c-9005-46a6-937d-9dab88cb2c6a\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " Sep 16 13:51:39.661401 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.661271 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-webhook-cert\") pod \"1596e27c-9005-46a6-937d-9dab88cb2c6a\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " Sep 16 13:51:39.661401 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.661296 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z7gv\" (UniqueName: \"kubernetes.io/projected/1596e27c-9005-46a6-937d-9dab88cb2c6a-kube-api-access-6z7gv\") pod \"1596e27c-9005-46a6-937d-9dab88cb2c6a\" (UID: \"1596e27c-9005-46a6-937d-9dab88cb2c6a\") " Sep 16 13:51:39.663396 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.663366 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1596e27c-9005-46a6-937d-9dab88cb2c6a" (UID: "1596e27c-9005-46a6-937d-9dab88cb2c6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:51:39.663527 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.663393 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1596e27c-9005-46a6-937d-9dab88cb2c6a" (UID: "1596e27c-9005-46a6-937d-9dab88cb2c6a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:51:39.663527 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.663460 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1596e27c-9005-46a6-937d-9dab88cb2c6a-kube-api-access-6z7gv" (OuterVolumeSpecName: "kube-api-access-6z7gv") pod "1596e27c-9005-46a6-937d-9dab88cb2c6a" (UID: "1596e27c-9005-46a6-937d-9dab88cb2c6a"). InnerVolumeSpecName "kube-api-access-6z7gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:51:39.762580 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.762506 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:51:39.762580 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.762532 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1596e27c-9005-46a6-937d-9dab88cb2c6a-webhook-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:51:39.762580 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:39.762541 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z7gv\" (UniqueName: \"kubernetes.io/projected/1596e27c-9005-46a6-937d-9dab88cb2c6a-kube-api-access-6z7gv\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:51:40.300533 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:40.300496 2581 generic.go:356] "Generic (PLEG): container finished" podID="1596e27c-9005-46a6-937d-9dab88cb2c6a" containerID="d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21" exitCode=0 Sep 16 13:51:40.300964 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:40.300545 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5664c89cd7-ltml9" Sep 16 13:51:40.300964 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:40.300578 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5664c89cd7-ltml9" event={"ID":"1596e27c-9005-46a6-937d-9dab88cb2c6a","Type":"ContainerDied","Data":"d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21"} Sep 16 13:51:40.300964 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:40.300632 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5664c89cd7-ltml9" event={"ID":"1596e27c-9005-46a6-937d-9dab88cb2c6a","Type":"ContainerDied","Data":"43eb85777ed6973263bb650976c8d818411acca97767a9ed7fbb4a66587f1c0f"} Sep 16 13:51:40.300964 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:40.300656 2581 scope.go:117] "RemoveContainer" containerID="d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21" Sep 16 13:51:40.309910 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:40.309894 2581 scope.go:117] "RemoveContainer" containerID="d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21" Sep 16 13:51:40.310240 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:51:40.310221 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21\": container with ID starting with d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21 not found: ID does not exist" containerID="d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21" Sep 16 13:51:40.310298 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:40.310249 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21"} err="failed to get container status \"d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21\": rpc error: code = NotFound desc = could not find container \"d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21\": container with ID starting with d8f4ba478fb1973e1572617bbd9bed1aec1a25c1a76e9ebf0235e2a6b796bd21 not found: ID does not exist" Sep 16 13:51:40.329399 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:40.329370 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5664c89cd7-ltml9"] Sep 16 13:51:40.331677 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:40.331658 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5664c89cd7-ltml9"] Sep 16 13:51:40.515999 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:51:40.515965 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1596e27c-9005-46a6-937d-9dab88cb2c6a" path="/var/lib/kubelet/pods/1596e27c-9005-46a6-937d-9dab88cb2c6a/volumes" Sep 16 13:52:02.573115 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.573079 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2"] Sep 16 13:52:02.573625 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.573489 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1596e27c-9005-46a6-937d-9dab88cb2c6a" containerName="manager" Sep 16 13:52:02.573625 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.573504 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1596e27c-9005-46a6-937d-9dab88cb2c6a" containerName="manager" Sep 16 13:52:02.573625 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.573587 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1596e27c-9005-46a6-937d-9dab88cb2c6a" containerName="manager" Sep 16 13:52:02.580182 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.580151 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.583461 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.583434 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 16 13:52:02.583650 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.583625 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 16 13:52:02.583789 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.583624 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kq7tn\"" Sep 16 13:52:02.585925 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.585905 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2"] Sep 16 13:52:02.658506 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.658468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-metrics-certs\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.658683 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.658535 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0461a8-b839-4c98-acf6-baed82905c86-openshift-service-ca\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.658683 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.658575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-maas-api-tls\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.658683 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.658649 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw9db\" (UniqueName: \"kubernetes.io/projected/2e0461a8-b839-4c98-acf6-baed82905c86-kube-api-access-cw9db\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.759496 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.759462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-metrics-certs\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.759659 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.759517 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0461a8-b839-4c98-acf6-baed82905c86-openshift-service-ca\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.759659 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.759552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-maas-api-tls\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.759659 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.759586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw9db\" (UniqueName: \"kubernetes.io/projected/2e0461a8-b839-4c98-acf6-baed82905c86-kube-api-access-cw9db\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.760280 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.760255 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0461a8-b839-4c98-acf6-baed82905c86-openshift-service-ca\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.761893 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.761872 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-maas-api-tls\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.761893 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.761890 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-metrics-certs\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.770326 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.770305 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw9db\" (UniqueName: \"kubernetes.io/projected/2e0461a8-b839-4c98-acf6-baed82905c86-kube-api-access-cw9db\") pod \"maas-api-557699c6fc-9h6d2\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:02.892914 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:02.892833 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:03.144084 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.144003 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4kn2r"] Sep 16 13:52:03.148569 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.148548 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" Sep 16 13:52:03.151212 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.151192 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tffrm\"" Sep 16 13:52:03.151295 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.151229 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 16 13:52:03.159984 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.159960 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4kn2r"] Sep 16 13:52:03.175426 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.175397 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2"] Sep 16 13:52:03.177074 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:52:03.177051 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0461a8_b839_4c98_acf6_baed82905c86.slice/crio-39d6288c03139fd87e10ca4321e7522bf8e3ad9707462ce0a1adfd45fba48bd5 WatchSource:0}: Error finding container 39d6288c03139fd87e10ca4321e7522bf8e3ad9707462ce0a1adfd45fba48bd5: Status 404 returned error can't find the container with id 39d6288c03139fd87e10ca4321e7522bf8e3ad9707462ce0a1adfd45fba48bd5 Sep 16 13:52:03.264755 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.264725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485872f2-06f0-40e9-99a9-f603cd536dd4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4kn2r\" (UID: \"485872f2-06f0-40e9-99a9-f603cd536dd4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" Sep 16 13:52:03.264933 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.264771 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg9sf\" (UniqueName: \"kubernetes.io/projected/485872f2-06f0-40e9-99a9-f603cd536dd4-kube-api-access-gg9sf\") pod \"payload-processing-5b54b7fbb7-4kn2r\" (UID: \"485872f2-06f0-40e9-99a9-f603cd536dd4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" Sep 16 13:52:03.365889 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.365852 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg9sf\" (UniqueName: \"kubernetes.io/projected/485872f2-06f0-40e9-99a9-f603cd536dd4-kube-api-access-gg9sf\") pod \"payload-processing-5b54b7fbb7-4kn2r\" (UID: \"485872f2-06f0-40e9-99a9-f603cd536dd4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" Sep 16 13:52:03.366055 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.365941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485872f2-06f0-40e9-99a9-f603cd536dd4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4kn2r\" (UID: \"485872f2-06f0-40e9-99a9-f603cd536dd4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" Sep 16 13:52:03.366616 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.366596 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485872f2-06f0-40e9-99a9-f603cd536dd4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4kn2r\" (UID: \"485872f2-06f0-40e9-99a9-f603cd536dd4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" Sep 16 13:52:03.375403 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.375380 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg9sf\" (UniqueName: \"kubernetes.io/projected/485872f2-06f0-40e9-99a9-f603cd536dd4-kube-api-access-gg9sf\") pod \"payload-processing-5b54b7fbb7-4kn2r\" (UID: \"485872f2-06f0-40e9-99a9-f603cd536dd4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" Sep 16 13:52:03.426179 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.426078 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" event={"ID":"2e0461a8-b839-4c98-acf6-baed82905c86","Type":"ContainerStarted","Data":"39d6288c03139fd87e10ca4321e7522bf8e3ad9707462ce0a1adfd45fba48bd5"} Sep 16 13:52:03.460185 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.460131 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" Sep 16 13:52:03.733415 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:03.733388 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4kn2r"] Sep 16 13:52:03.735192 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:52:03.735162 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485872f2_06f0_40e9_99a9_f603cd536dd4.slice/crio-22533563874c7f42a02669b51c3699368caa67207a3e7a7a4c408affc588c755 WatchSource:0}: Error finding container 22533563874c7f42a02669b51c3699368caa67207a3e7a7a4c408affc588c755: Status 404 returned error can't find the container with id 22533563874c7f42a02669b51c3699368caa67207a3e7a7a4c408affc588c755 Sep 16 13:52:04.436457 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:04.436411 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" event={"ID":"485872f2-06f0-40e9-99a9-f603cd536dd4","Type":"ContainerStarted","Data":"22533563874c7f42a02669b51c3699368caa67207a3e7a7a4c408affc588c755"} Sep 16 13:52:05.444587 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:05.444534 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" event={"ID":"2e0461a8-b839-4c98-acf6-baed82905c86","Type":"ContainerStarted","Data":"8ae7af8c577fe5b0bd80a2be097237b24f022ec98d630db5f14a881d9d845ad3"} Sep 16 13:52:05.445036 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:05.444628 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:05.446797 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:05.446766 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" event={"ID":"485872f2-06f0-40e9-99a9-f603cd536dd4","Type":"ContainerStarted","Data":"7ac70a4c4e21e7b948479fad6d00a082270d8f45eb94b6f6ae29bf6522a7dd53"} Sep 16 13:52:05.446936 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:05.446816 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" Sep 16 13:52:05.465696 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:05.465648 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" podStartSLOduration=1.341151253 podStartE2EDuration="3.465629881s" totalImagesPullingTime="2.124478628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:05.462388471 +0000 UTC m=+559.425747297" watchObservedRunningTime="2026-09-16 13:52:05.465629881 +0000 UTC m=+559.428988994" Sep 16 13:52:05.491734 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:05.491690 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" podStartSLOduration=0.923587966 podStartE2EDuration="2.491677769s" totalImagesPullingTime="1.568089803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:05.489197505 +0000 UTC m=+559.452556333" watchObservedRunningTime="2026-09-16 13:52:05.491677769 +0000 UTC m=+559.455036595" Sep 16 13:52:11.458546 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:11.458512 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 13:52:16.454330 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:16.454295 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4kn2r" Sep 16 13:52:32.345310 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.345273 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf"] Sep 16 13:52:32.349043 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.349024 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.351601 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.351576 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 13:52:32.352488 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.352467 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 13:52:32.352590 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.352467 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 16 13:52:32.352590 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.352504 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g67b7\"" Sep 16 13:52:32.357725 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.357702 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf"] Sep 16 13:52:32.502914 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.502865 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.503195 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.502960 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.503195 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.503015 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.503195 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.503038 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.503195 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.503082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6lt\" (UniqueName: \"kubernetes.io/projected/41562f83-cf4e-4b0d-8a0f-250683374aa7-kube-api-access-md6lt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.503443 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.503200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41562f83-cf4e-4b0d-8a0f-250683374aa7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.604588 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.604511 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.604588 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.604558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.604588 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.604580 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.604814 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.604694 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6lt\" (UniqueName: \"kubernetes.io/projected/41562f83-cf4e-4b0d-8a0f-250683374aa7-kube-api-access-md6lt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.604814 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.604734 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41562f83-cf4e-4b0d-8a0f-250683374aa7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.604814 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.604793 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.605014 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.604986 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.605014 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.605009 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.605254 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.605191 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.606765 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.606743 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41562f83-cf4e-4b0d-8a0f-250683374aa7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.607056 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.607040 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41562f83-cf4e-4b0d-8a0f-250683374aa7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.614510 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.614471 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6lt\" (UniqueName: \"kubernetes.io/projected/41562f83-cf4e-4b0d-8a0f-250683374aa7-kube-api-access-md6lt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf\" (UID: \"41562f83-cf4e-4b0d-8a0f-250683374aa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.660898 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.660872 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:32.936853 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:32.936764 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf"] Sep 16 13:52:32.941177 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:52:32.941131 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41562f83_cf4e_4b0d_8a0f_250683374aa7.slice/crio-137ef09b261e7daff4a256f0448de663c0e654c134ecebbe9fe6a28c682bf565 WatchSource:0}: Error finding container 137ef09b261e7daff4a256f0448de663c0e654c134ecebbe9fe6a28c682bf565: Status 404 returned error can't find the container with id 137ef09b261e7daff4a256f0448de663c0e654c134ecebbe9fe6a28c682bf565 Sep 16 13:52:33.601176 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:33.601117 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" event={"ID":"41562f83-cf4e-4b0d-8a0f-250683374aa7","Type":"ContainerStarted","Data":"137ef09b261e7daff4a256f0448de663c0e654c134ecebbe9fe6a28c682bf565"} Sep 16 13:52:34.608622 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:34.608586 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" event={"ID":"41562f83-cf4e-4b0d-8a0f-250683374aa7","Type":"ContainerStarted","Data":"002983d0700e3418b0662a5005ef2defe74f0a814e87cf741c0584ab3b357d4d"} Sep 16 13:52:34.626433 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:34.626384 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" podStartSLOduration=1.626973679 podStartE2EDuration="2.626367977s" totalImagesPullingTime="999.394298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:34.625994909 +0000 UTC m=+588.589353735" watchObservedRunningTime="2026-09-16 13:52:34.626367977 +0000 UTC m=+588.589726804" Sep 16 13:52:42.661399 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:42.661365 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:42.661764 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:42.661435 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:42.673913 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:42.673889 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:43.430592 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.430560 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b"] Sep 16 13:52:43.434180 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.434155 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.436627 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.436605 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 16 13:52:43.442900 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.442879 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b"] Sep 16 13:52:43.595805 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.595773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.595946 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.595809 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.595946 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.595844 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4sd\" (UniqueName: \"kubernetes.io/projected/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-kube-api-access-gd4sd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.595946 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.595868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.595946 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.595883 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.595946 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.595900 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.668289 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.668263 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf" Sep 16 13:52:43.697512 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.697429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.697512 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.697471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.697724 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.697524 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4sd\" (UniqueName: \"kubernetes.io/projected/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-kube-api-access-gd4sd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.697724 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.697545 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.697724 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.697566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.697724 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.697593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.697941 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.697919 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.698236 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.698202 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.698361 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.698285 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.699778 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.699756 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.700501 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.700473 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.706450 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.706427 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4sd\" (UniqueName: \"kubernetes.io/projected/5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf-kube-api-access-gd4sd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b\" (UID: \"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:43.746845 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:43.746822 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:44.019276 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:44.019251 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b"] Sep 16 13:52:44.020474 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:52:44.020446 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5389af03_dc57_4ecc_a4ef_bc3fbb85ebbf.slice/crio-91e160c9f0bd9078b4af524ebe56cca5be0bd89010f0c28f11d4b23eaedcd207 WatchSource:0}: Error finding container 91e160c9f0bd9078b4af524ebe56cca5be0bd89010f0c28f11d4b23eaedcd207: Status 404 returned error can't find the container with id 91e160c9f0bd9078b4af524ebe56cca5be0bd89010f0c28f11d4b23eaedcd207 Sep 16 13:52:44.664221 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:44.664188 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" event={"ID":"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf","Type":"ContainerStarted","Data":"3a1bf674e9f9a0cd1dda9f01a869ede93b95e9f4aed3f79b0a78a72aafcfda98"} Sep 16 13:52:44.664392 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:44.664237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" event={"ID":"5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf","Type":"ContainerStarted","Data":"91e160c9f0bd9078b4af524ebe56cca5be0bd89010f0c28f11d4b23eaedcd207"} Sep 16 13:52:44.693578 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:44.693533 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" podStartSLOduration=1.5052486809999999 podStartE2EDuration="1.693518556s" totalImagesPullingTime="188.269875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:44.690897829 +0000 UTC m=+598.654256657" watchObservedRunningTime="2026-09-16 13:52:44.693518556 +0000 UTC m=+598.656877387" Sep 16 13:52:46.503960 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:46.503935 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 13:52:46.505293 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:46.505268 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 13:52:46.514470 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:46.514444 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:52:46.515104 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:46.515079 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:52:53.747690 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:53.747660 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:53.748061 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:53.747711 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:53.760123 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:53.760099 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:52:54.728571 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:52:54.728541 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b" Sep 16 13:53:06.642989 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.642954 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd"] Sep 16 13:53:06.646843 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.646814 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.649500 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.649479 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 16 13:53:06.658953 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.658932 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd"] Sep 16 13:53:06.672411 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.672366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wjks\" (UniqueName: \"kubernetes.io/projected/1012cac1-5ca3-4865-8b18-dec6624176e4-kube-api-access-4wjks\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.672541 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.672436 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.672541 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.672487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.672541 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.672511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.672761 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.672577 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.672761 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.672623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1012cac1-5ca3-4865-8b18-dec6624176e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.773795 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.773763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.773795 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.773797 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.774026 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.773826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.774026 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.773946 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1012cac1-5ca3-4865-8b18-dec6624176e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.774130 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.774057 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wjks\" (UniqueName: \"kubernetes.io/projected/1012cac1-5ca3-4865-8b18-dec6624176e4-kube-api-access-4wjks\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.774130 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.774110 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.774310 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.774289 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.774375 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.774308 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.774375 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.774353 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.776756 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.776731 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1012cac1-5ca3-4865-8b18-dec6624176e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.777581 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.777563 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1012cac1-5ca3-4865-8b18-dec6624176e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.786535 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.786512 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wjks\" (UniqueName: \"kubernetes.io/projected/1012cac1-5ca3-4865-8b18-dec6624176e4-kube-api-access-4wjks\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fw2sd\" (UID: \"1012cac1-5ca3-4865-8b18-dec6624176e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:06.961100 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:06.961072 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:07.236557 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:07.236530 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd"] Sep 16 13:53:07.237216 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:53:07.237182 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1012cac1_5ca3_4865_8b18_dec6624176e4.slice/crio-12b1f9179021c7ed36fe4cccf8a7da9d2d902fcabf0a8b32c1fa74fc12a30cc3 WatchSource:0}: Error finding container 12b1f9179021c7ed36fe4cccf8a7da9d2d902fcabf0a8b32c1fa74fc12a30cc3: Status 404 returned error can't find the container with id 12b1f9179021c7ed36fe4cccf8a7da9d2d902fcabf0a8b32c1fa74fc12a30cc3 Sep 16 13:53:07.795404 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:07.795367 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" event={"ID":"1012cac1-5ca3-4865-8b18-dec6624176e4","Type":"ContainerStarted","Data":"303b9e05f21b9fb2b0223feedb2814a0139856f374372adb21ac6f4675ac8ddf"} Sep 16 13:53:07.795824 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:07.795423 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" event={"ID":"1012cac1-5ca3-4865-8b18-dec6624176e4","Type":"ContainerStarted","Data":"12b1f9179021c7ed36fe4cccf8a7da9d2d902fcabf0a8b32c1fa74fc12a30cc3"} Sep 16 13:53:07.821408 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:07.821359 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" podStartSLOduration=1.633453617 podStartE2EDuration="1.821345429s" totalImagesPullingTime="187.891812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:53:07.818992126 +0000 UTC m=+621.782350954" watchObservedRunningTime="2026-09-16 13:53:07.821345429 +0000 UTC m=+621.784704255" Sep 16 13:53:16.961902 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:16.961809 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:16.961902 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:16.961880 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:16.974400 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:16.974378 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:53:17.860236 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:53:17.860210 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fw2sd" Sep 16 13:55:43.434649 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.434603 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj"] Sep 16 13:55:43.438527 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.438505 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.441035 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.441014 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-035c64-openshift-default-dockercfg-ljhsx\"" Sep 16 13:55:43.454321 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.454298 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj"] Sep 16 13:55:43.600380 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.600342 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-credential-socket\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.600553 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.600396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-socket\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.600553 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.600415 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-certs\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.600553 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.600433 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n69wk\" (UniqueName: \"kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-kube-api-access-n69wk\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.600697 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.600568 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34473024-9c65-41aa-9007-5ddbc65490eb-istio-podinfo\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.600697 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.600625 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-data\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.600697 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.600669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-istio-token\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.600697 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.600687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34473024-9c65-41aa-9007-5ddbc65490eb-istiod-ca-cert\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.600844 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.600742 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-envoy\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702022 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.701947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-istio-token\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702022 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.701979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34473024-9c65-41aa-9007-5ddbc65490eb-istiod-ca-cert\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702022 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-envoy\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702340 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-credential-socket\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702340 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-socket\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702340 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702095 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-certs\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702340 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702121 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69wk\" (UniqueName: \"kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-kube-api-access-n69wk\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702340 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34473024-9c65-41aa-9007-5ddbc65490eb-istio-podinfo\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702340 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-data\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702535 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-credential-socket\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702600 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-certs\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702774 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702739 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-data\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702774 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702764 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34473024-9c65-41aa-9007-5ddbc65490eb-istiod-ca-cert\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.702885 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.702844 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-socket\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.704164 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.704129 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-envoy\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.704502 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.704483 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34473024-9c65-41aa-9007-5ddbc65490eb-istio-podinfo\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.712779 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.712756 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-istio-token\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.717660 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.717637 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69wk\" (UniqueName: \"kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-kube-api-access-n69wk\") pod \"e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:43.749622 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:43.749591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:44.049210 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:55:44.049169 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34473024_9c65_41aa_9007_5ddbc65490eb.slice/crio-186f874aa112cb2f51ba1708d47da97b0f0b1a8526d5584022282de88bda1b34 WatchSource:0}: Error finding container 186f874aa112cb2f51ba1708d47da97b0f0b1a8526d5584022282de88bda1b34: Status 404 returned error can't find the container with id 186f874aa112cb2f51ba1708d47da97b0f0b1a8526d5584022282de88bda1b34 Sep 16 13:55:44.049357 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:44.049299 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj"] Sep 16 13:55:44.051281 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:44.051264 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:55:44.678434 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:44.678393 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" event={"ID":"34473024-9c65-41aa-9007-5ddbc65490eb","Type":"ContainerStarted","Data":"186f874aa112cb2f51ba1708d47da97b0f0b1a8526d5584022282de88bda1b34"} Sep 16 13:55:47.004306 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.004271 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 13:55:47.004544 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.004342 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 13:55:47.004544 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.004371 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 13:55:47.666740 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.666710 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t"] Sep 16 13:55:47.670468 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.670445 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.674365 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.674341 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-19203d-serving-cert\"" Sep 16 13:55:47.688404 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.688383 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t"] Sep 16 13:55:47.701184 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.701135 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" event={"ID":"34473024-9c65-41aa-9007-5ddbc65490eb","Type":"ContainerStarted","Data":"f70d5ca339cd189c088c91a92dca490e4bd5b91f8c4ca32206e97f66eb4c6aea"} Sep 16 13:55:47.741230 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.741200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-metrics-certs\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.741351 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.741258 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae5bcca-86ee-43b0-b0d6-b09529a19152-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.741476 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.741401 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68j9m\" (UniqueName: \"kubernetes.io/projected/6ae5bcca-86ee-43b0-b0d6-b09529a19152-kube-api-access-68j9m\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.741591 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.741510 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-maas-api-tls\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.750025 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.750006 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:47.765237 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.765190 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" podStartSLOduration=1.8126485639999999 podStartE2EDuration="4.765169511s" totalImagesPullingTime="2.952520947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:47.764450487 +0000 UTC m=+781.727809349" watchObservedRunningTime="2026-09-16 13:55:47.765169511 +0000 UTC m=+781.728528356" Sep 16 13:55:47.843024 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.842988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68j9m\" (UniqueName: \"kubernetes.io/projected/6ae5bcca-86ee-43b0-b0d6-b09529a19152-kube-api-access-68j9m\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.843223 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.843035 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-maas-api-tls\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.843223 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.843171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-metrics-certs\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.843360 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.843221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae5bcca-86ee-43b0-b0d6-b09529a19152-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.843360 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:55:47.843222 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-serving-cert: secret "maas-api-e2e-worker-w4-19203d-serving-cert" not found Sep 16 13:55:47.843360 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:55:47.843352 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-maas-api-tls podName:6ae5bcca-86ee-43b0-b0d6-b09529a19152 nodeName:}" failed. No retries permitted until 2026-09-16 13:55:48.343329714 +0000 UTC m=+782.306688522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-maas-api-tls") pod "maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" (UID: "6ae5bcca-86ee-43b0-b0d6-b09529a19152") : secret "maas-api-e2e-worker-w4-19203d-serving-cert" not found Sep 16 13:55:47.843786 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.843765 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae5bcca-86ee-43b0-b0d6-b09529a19152-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.845873 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.845848 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-metrics-certs\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:47.862037 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:47.862007 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68j9m\" (UniqueName: \"kubernetes.io/projected/6ae5bcca-86ee-43b0-b0d6-b09529a19152-kube-api-access-68j9m\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:48.349501 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:48.349467 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-maas-api-tls\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:48.351727 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:48.351696 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-maas-api-tls\") pod \"maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:48.584184 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:48.584121 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:48.755226 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:48.755198 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:48.869996 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:48.869969 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t"] Sep 16 13:55:48.870839 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:55:48.870816 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae5bcca_86ee_43b0_b0d6_b09529a19152.slice/crio-43cb0e64f6c1f6abe1bf03f8cf062958bca26d568722a625622066070314463d WatchSource:0}: Error finding container 43cb0e64f6c1f6abe1bf03f8cf062958bca26d568722a625622066070314463d: Status 404 returned error can't find the container with id 43cb0e64f6c1f6abe1bf03f8cf062958bca26d568722a625622066070314463d Sep 16 13:55:49.163219 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.163181 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6"] Sep 16 13:55:49.168101 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.168079 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.170539 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.170520 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b52a89-serving-cert\"" Sep 16 13:55:49.186852 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.186824 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6"] Sep 16 13:55:49.258742 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.258708 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-metrics-certs\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.258742 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.258746 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-maas-api-tls\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.258996 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.258873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829kp\" (UniqueName: \"kubernetes.io/projected/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-kube-api-access-829kp\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.258996 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.258945 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.360071 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.360044 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829kp\" (UniqueName: \"kubernetes.io/projected/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-kube-api-access-829kp\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.360496 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.360160 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.360496 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.360219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-metrics-certs\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.360496 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.360243 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-maas-api-tls\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.360496 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:55:49.360385 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-serving-cert: secret "maas-api-e2e-inf-a-b52a89-serving-cert" not found Sep 16 13:55:49.360496 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:55:49.360432 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-maas-api-tls podName:4dff28b8-676b-44f7-8d39-5a7d25a4bc28 nodeName:}" failed. No retries permitted until 2026-09-16 13:55:49.860417386 +0000 UTC m=+783.823776191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-maas-api-tls") pod "maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" (UID: "4dff28b8-676b-44f7-8d39-5a7d25a4bc28") : secret "maas-api-e2e-inf-a-b52a89-serving-cert" not found Sep 16 13:55:49.360761 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.360740 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.362754 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.362736 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-metrics-certs\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.372734 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.372711 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-829kp\" (UniqueName: \"kubernetes.io/projected/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-kube-api-access-829kp\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.714520 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.714477 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" event={"ID":"6ae5bcca-86ee-43b0-b0d6-b09529a19152","Type":"ContainerStarted","Data":"82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900"} Sep 16 13:55:49.714698 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.714667 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:49.714770 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.714708 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" event={"ID":"6ae5bcca-86ee-43b0-b0d6-b09529a19152","Type":"ContainerStarted","Data":"43cb0e64f6c1f6abe1bf03f8cf062958bca26d568722a625622066070314463d"} Sep 16 13:55:49.714926 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.714907 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:49.715857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.715835 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:55:49.748393 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.748351 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" podStartSLOduration=2.27204761 podStartE2EDuration="2.74833792s" totalImagesPullingTime="476.29031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:49.745786162 +0000 UTC m=+783.709144988" watchObservedRunningTime="2026-09-16 13:55:49.74833792 +0000 UTC m=+783.711696747" Sep 16 13:55:49.866486 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.866453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-maas-api-tls\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:49.868942 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:49.868920 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-maas-api-tls\") pod \"maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:50.079363 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:50.079282 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:50.395485 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:50.395457 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6"] Sep 16 13:55:50.397125 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:55:50.397096 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dff28b8_676b_44f7_8d39_5a7d25a4bc28.slice/crio-5c82c7c6254bd1996267cb80c1f17a4ae88faa4716ed9db363a44637c6172311 WatchSource:0}: Error finding container 5c82c7c6254bd1996267cb80c1f17a4ae88faa4716ed9db363a44637c6172311: Status 404 returned error can't find the container with id 5c82c7c6254bd1996267cb80c1f17a4ae88faa4716ed9db363a44637c6172311 Sep 16 13:55:50.721208 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:50.721172 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" event={"ID":"4dff28b8-676b-44f7-8d39-5a7d25a4bc28","Type":"ContainerStarted","Data":"5c82c7c6254bd1996267cb80c1f17a4ae88faa4716ed9db363a44637c6172311"} Sep 16 13:55:51.729348 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:51.729308 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" event={"ID":"4dff28b8-676b-44f7-8d39-5a7d25a4bc28","Type":"ContainerStarted","Data":"5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2"} Sep 16 13:55:51.751022 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:51.750967 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" podStartSLOduration=2.434642361 podStartE2EDuration="2.750950751s" totalImagesPullingTime="316.30839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:51.750248222 +0000 UTC m=+785.713607077" watchObservedRunningTime="2026-09-16 13:55:51.750950751 +0000 UTC m=+785.714309587" Sep 16 13:55:52.735503 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:52.735472 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:55:53.864284 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:53.864253 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg"] Sep 16 13:55:53.867984 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:53.867960 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:53.871295 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:53.871271 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-035c64-serving-cert\"" Sep 16 13:55:53.881041 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:53.881016 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg"] Sep 16 13:55:53.907135 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:53.907091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-maas-api-tls\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:53.907288 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:53.907182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71540276-4d70-40ce-a0b6-7d54fa468543-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:53.907288 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:53.907250 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvfmv\" (UniqueName: \"kubernetes.io/projected/71540276-4d70-40ce-a0b6-7d54fa468543-kube-api-access-bvfmv\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:53.907288 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:53.907285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-metrics-certs\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:54.008264 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:54.008224 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-metrics-certs\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:54.009090 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:54.009063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-maas-api-tls\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:54.009337 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:54.009315 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71540276-4d70-40ce-a0b6-7d54fa468543-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:54.009537 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:54.009521 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvfmv\" (UniqueName: \"kubernetes.io/projected/71540276-4d70-40ce-a0b6-7d54fa468543-kube-api-access-bvfmv\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:54.010035 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:55:54.010012 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-serving-cert: secret "maas-api-e2e-worker-w0-035c64-serving-cert" not found Sep 16 13:55:54.010267 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:55:54.010242 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-maas-api-tls podName:71540276-4d70-40ce-a0b6-7d54fa468543 nodeName:}" failed. No retries permitted until 2026-09-16 13:55:54.510212511 +0000 UTC m=+788.473571681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-maas-api-tls") pod "maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" (UID: "71540276-4d70-40ce-a0b6-7d54fa468543") : secret "maas-api-e2e-worker-w0-035c64-serving-cert" not found Sep 16 13:55:54.011265 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:54.011244 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71540276-4d70-40ce-a0b6-7d54fa468543-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:54.014131 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:54.014108 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-metrics-certs\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:54.025415 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:54.025394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvfmv\" (UniqueName: \"kubernetes.io/projected/71540276-4d70-40ce-a0b6-7d54fa468543-kube-api-access-bvfmv\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:54.513289 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:54.513256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-maas-api-tls\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:54.516754 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:54.516673 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-maas-api-tls\") pod \"maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:54.779501 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:54.779412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:55.069906 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:55.069870 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg"] Sep 16 13:55:55.071219 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:55:55.071186 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71540276_4d70_40ce_a0b6_7d54fa468543.slice/crio-f748f09cfac78b856ae78f6ed6259c6658061a40e0cf09666311a9e515e1b300 WatchSource:0}: Error finding container f748f09cfac78b856ae78f6ed6259c6658061a40e0cf09666311a9e515e1b300: Status 404 returned error can't find the container with id f748f09cfac78b856ae78f6ed6259c6658061a40e0cf09666311a9e515e1b300 Sep 16 13:55:55.726700 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:55.726670 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 13:55:55.754696 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:55.754655 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" event={"ID":"71540276-4d70-40ce-a0b6-7d54fa468543","Type":"ContainerStarted","Data":"711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f"} Sep 16 13:55:55.754850 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:55.754712 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" event={"ID":"71540276-4d70-40ce-a0b6-7d54fa468543","Type":"ContainerStarted","Data":"f748f09cfac78b856ae78f6ed6259c6658061a40e0cf09666311a9e515e1b300"} Sep 16 13:55:55.754850 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:55.754831 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:55:55.775148 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:55.775093 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" podStartSLOduration=2.452897008 podStartE2EDuration="2.77507758s" totalImagesPullingTime="322.180572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:55.771942636 +0000 UTC m=+789.735301466" watchObservedRunningTime="2026-09-16 13:55:55.77507758 +0000 UTC m=+789.738436407" Sep 16 13:55:58.746907 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:55:58.746876 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 13:56:00.266982 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.266945 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj"] Sep 16 13:56:00.272375 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.272348 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.274785 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.274756 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-617597-serving-cert\"" Sep 16 13:56:00.283621 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.283596 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj"] Sep 16 13:56:00.368618 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.368578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-openshift-service-ca\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.368618 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.368622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbpmb\" (UniqueName: \"kubernetes.io/projected/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-kube-api-access-jbpmb\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.368857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.368643 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-maas-api-tls\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.368857 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.368748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-metrics-certs\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.469809 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.469771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-openshift-service-ca\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.469809 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.469811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbpmb\" (UniqueName: \"kubernetes.io/projected/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-kube-api-access-jbpmb\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.470060 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.469832 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-maas-api-tls\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.470060 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.469865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-metrics-certs\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.470667 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.470642 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-openshift-service-ca\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.472301 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.472272 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-maas-api-tls\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.472407 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.472351 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-metrics-certs\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.479994 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.479968 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbpmb\" (UniqueName: \"kubernetes.io/projected/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-kube-api-access-jbpmb\") pod \"maas-api-e2e-autores-617597-5b6877bf86-nxbnj\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.585772 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.585707 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:00.868602 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:00.868575 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj"] Sep 16 13:56:00.870032 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:56:00.870006 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25215817_6cd2_4ea9_a239_c7c7a7d5bf10.slice/crio-31fe0ee3d506761cefd62ca45ac1a9dea1c148a49973930140445c98d09cb6f0 WatchSource:0}: Error finding container 31fe0ee3d506761cefd62ca45ac1a9dea1c148a49973930140445c98d09cb6f0: Status 404 returned error can't find the container with id 31fe0ee3d506761cefd62ca45ac1a9dea1c148a49973930140445c98d09cb6f0 Sep 16 13:56:01.428326 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.428291 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg"] Sep 16 13:56:01.433512 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.433489 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.436184 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.436160 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1dcc64e3-serving-cert\"" Sep 16 13:56:01.443887 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.443863 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg"] Sep 16 13:56:01.481244 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.481203 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.481405 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.481325 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa50451-5e01-4e4c-9a90-ed7a03939e09-openshift-service-ca\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.481472 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.481456 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4blln\" (UniqueName: \"kubernetes.io/projected/bfa50451-5e01-4e4c-9a90-ed7a03939e09-kube-api-access-4blln\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.481508 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.481490 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-metrics-certs\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.582797 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.582771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.582943 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.582885 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa50451-5e01-4e4c-9a90-ed7a03939e09-openshift-service-ca\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.583011 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:56:01.582957 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-serving-cert: secret "maas-api-e2e-ait-1dcc64e3-serving-cert" not found Sep 16 13:56:01.583062 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:56:01.583031 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls podName:bfa50451-5e01-4e4c-9a90-ed7a03939e09 nodeName:}" failed. No retries permitted until 2026-09-16 13:56:02.083010294 +0000 UTC m=+796.046369103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls") pod "maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" (UID: "bfa50451-5e01-4e4c-9a90-ed7a03939e09") : secret "maas-api-e2e-ait-1dcc64e3-serving-cert" not found Sep 16 13:56:01.583122 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.582964 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4blln\" (UniqueName: \"kubernetes.io/projected/bfa50451-5e01-4e4c-9a90-ed7a03939e09-kube-api-access-4blln\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.583196 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.583168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-metrics-certs\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.583621 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.583600 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa50451-5e01-4e4c-9a90-ed7a03939e09-openshift-service-ca\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.585497 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.585479 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-metrics-certs\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.601921 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.601890 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4blln\" (UniqueName: \"kubernetes.io/projected/bfa50451-5e01-4e4c-9a90-ed7a03939e09-kube-api-access-4blln\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:01.766166 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.766109 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 13:56:01.792360 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.792325 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" event={"ID":"25215817-6cd2-4ea9-a239-c7c7a7d5bf10","Type":"ContainerStarted","Data":"5279c6954c3683f6b6f06a9294c11183a7a2922b2fb5634e5cf0ad0eafba3322"} Sep 16 13:56:01.792547 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.792379 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" event={"ID":"25215817-6cd2-4ea9-a239-c7c7a7d5bf10","Type":"ContainerStarted","Data":"31fe0ee3d506761cefd62ca45ac1a9dea1c148a49973930140445c98d09cb6f0"} Sep 16 13:56:01.792547 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.792410 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:01.876483 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:01.876419 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" podStartSLOduration=1.221448358 podStartE2EDuration="1.876396003s" totalImagesPullingTime="654.947645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:01.87529396 +0000 UTC m=+795.838652796" watchObservedRunningTime="2026-09-16 13:56:01.876396003 +0000 UTC m=+795.839754832" Sep 16 13:56:02.088748 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:02.088663 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:02.088896 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:56:02.088808 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-serving-cert: secret "maas-api-e2e-ait-1dcc64e3-serving-cert" not found Sep 16 13:56:02.088896 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:56:02.088876 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls podName:bfa50451-5e01-4e4c-9a90-ed7a03939e09 nodeName:}" failed. No retries permitted until 2026-09-16 13:56:03.088861675 +0000 UTC m=+797.052220480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls") pod "maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" (UID: "bfa50451-5e01-4e4c-9a90-ed7a03939e09") : secret "maas-api-e2e-ait-1dcc64e3-serving-cert" not found Sep 16 13:56:03.099386 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:03.099354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:03.101705 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:03.101686 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls\") pod \"maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:03.250643 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:03.250613 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:03.540745 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:03.540706 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg"] Sep 16 13:56:03.545588 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:56:03.545551 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa50451_5e01_4e4c_9a90_ed7a03939e09.slice/crio-d489ab03ddeb79cf4e04046198b89fcfee3409d2f70f2b579fe3ff7938b9f6b9 WatchSource:0}: Error finding container d489ab03ddeb79cf4e04046198b89fcfee3409d2f70f2b579fe3ff7938b9f6b9: Status 404 returned error can't find the container with id d489ab03ddeb79cf4e04046198b89fcfee3409d2f70f2b579fe3ff7938b9f6b9 Sep 16 13:56:03.816707 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:03.816609 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" event={"ID":"bfa50451-5e01-4e4c-9a90-ed7a03939e09","Type":"ContainerStarted","Data":"d489ab03ddeb79cf4e04046198b89fcfee3409d2f70f2b579fe3ff7938b9f6b9"} Sep 16 13:56:04.824451 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:04.824417 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" event={"ID":"bfa50451-5e01-4e4c-9a90-ed7a03939e09","Type":"ContainerStarted","Data":"2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33"} Sep 16 13:56:04.824926 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:04.824554 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:04.851032 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:04.850981 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" podStartSLOduration=3.300812504 podStartE2EDuration="3.850965885s" totalImagesPullingTime="550.153381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:04.849865683 +0000 UTC m=+798.813224491" watchObservedRunningTime="2026-09-16 13:56:04.850965885 +0000 UTC m=+798.814324711" Sep 16 13:56:07.810517 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:07.810481 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:56:10.842284 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:10.842210 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:11.687029 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.686994 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk"] Sep 16 13:56:11.693047 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.693021 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.695759 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.695732 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-95743e-serving-cert\"" Sep 16 13:56:11.712952 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.712855 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk"] Sep 16 13:56:11.796688 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.796653 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643f0199-dbb5-42bb-9490-d3118ec8d553-openshift-service-ca\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.796946 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.796921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-maas-api-tls\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.797100 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.797083 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-metrics-certs\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.797344 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.797324 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjvh\" (UniqueName: \"kubernetes.io/projected/643f0199-dbb5-42bb-9490-d3118ec8d553-kube-api-access-xmjvh\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.899081 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.899042 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-maas-api-tls\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.899579 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.899100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-metrics-certs\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.899579 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.899213 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjvh\" (UniqueName: \"kubernetes.io/projected/643f0199-dbb5-42bb-9490-d3118ec8d553-kube-api-access-xmjvh\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.899579 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.899349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643f0199-dbb5-42bb-9490-d3118ec8d553-openshift-service-ca\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.900199 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.900172 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643f0199-dbb5-42bb-9490-d3118ec8d553-openshift-service-ca\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.900313 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:56:11.900184 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-serving-cert: secret "maas-api-e2e-inf-b-95743e-serving-cert" not found Sep 16 13:56:11.900313 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:56:11.900285 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-maas-api-tls podName:643f0199-dbb5-42bb-9490-d3118ec8d553 nodeName:}" failed. No retries permitted until 2026-09-16 13:56:12.400264451 +0000 UTC m=+806.363623259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-maas-api-tls") pod "maas-api-e2e-inf-b-95743e-585c96497-tkjdk" (UID: "643f0199-dbb5-42bb-9490-d3118ec8d553") : secret "maas-api-e2e-inf-b-95743e-serving-cert" not found Sep 16 13:56:11.904673 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.904644 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-metrics-certs\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:11.909321 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:11.909294 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjvh\" (UniqueName: \"kubernetes.io/projected/643f0199-dbb5-42bb-9490-d3118ec8d553-kube-api-access-xmjvh\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:12.404822 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:12.404639 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-maas-api-tls\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:12.407842 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:12.407822 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-maas-api-tls\") pod \"maas-api-e2e-inf-b-95743e-585c96497-tkjdk\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:12.617761 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:12.617731 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:12.911998 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:12.911964 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk"] Sep 16 13:56:12.914659 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:56:12.914614 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643f0199_dbb5_42bb_9490_d3118ec8d553.slice/crio-8084b20b1c297d699f3460d6e79a75a8d34448bdd928ac282a869a0dff4aca1f WatchSource:0}: Error finding container 8084b20b1c297d699f3460d6e79a75a8d34448bdd928ac282a869a0dff4aca1f: Status 404 returned error can't find the container with id 8084b20b1c297d699f3460d6e79a75a8d34448bdd928ac282a869a0dff4aca1f Sep 16 13:56:13.879950 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:13.879910 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" event={"ID":"643f0199-dbb5-42bb-9490-d3118ec8d553","Type":"ContainerStarted","Data":"9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf"} Sep 16 13:56:13.880177 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:13.879978 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" event={"ID":"643f0199-dbb5-42bb-9490-d3118ec8d553","Type":"ContainerStarted","Data":"8084b20b1c297d699f3460d6e79a75a8d34448bdd928ac282a869a0dff4aca1f"} Sep 16 13:56:13.880177 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:13.880030 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:13.935182 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:13.935096 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" podStartSLOduration=2.593023432 podStartE2EDuration="2.935075686s" totalImagesPullingTime="342.052254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:13.931062835 +0000 UTC m=+807.894421661" watchObservedRunningTime="2026-09-16 13:56:13.935075686 +0000 UTC m=+807.898434513" Sep 16 13:56:16.906082 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:16.906049 2581 generic.go:356] "Generic (PLEG): container finished" podID="34473024-9c65-41aa-9007-5ddbc65490eb" containerID="f70d5ca339cd189c088c91a92dca490e4bd5b91f8c4ca32206e97f66eb4c6aea" exitCode=0 Sep 16 13:56:16.906499 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:16.906121 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" event={"ID":"34473024-9c65-41aa-9007-5ddbc65490eb","Type":"ContainerDied","Data":"f70d5ca339cd189c088c91a92dca490e4bd5b91f8c4ca32206e97f66eb4c6aea"} Sep 16 13:56:16.906570 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:16.906535 2581 scope.go:117] "RemoveContainer" containerID="f70d5ca339cd189c088c91a92dca490e4bd5b91f8c4ca32206e97f66eb4c6aea" Sep 16 13:56:16.907665 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:16.907635 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 13:56:16.907734 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:16.907707 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 13:56:16.907772 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:16.907754 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 13:56:17.915320 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:17.915286 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" event={"ID":"34473024-9c65-41aa-9007-5ddbc65490eb","Type":"ContainerStarted","Data":"50db27b94738fea29b0425aa5dac6937b77c654fbc724db9cc1a81461380abc4"} Sep 16 13:56:18.750320 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:18.750289 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:56:18.751551 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:18.751525 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 16 13:56:18.751675 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:18.751573 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 16 13:56:19.714701 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:19.714662 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:56:19.750817 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:19.750790 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 16 13:56:19.750963 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:19.750838 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 16 13:56:19.900573 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:19.900533 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 13:56:20.750130 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:20.750088 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 16 13:56:20.750640 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:20.750179 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 16 13:56:21.750559 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:21.750519 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 16 13:56:21.750925 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:21.750581 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 16 13:56:22.750263 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:22.750230 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 16 13:56:22.750436 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:22.750284 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 16 13:56:23.753974 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:23.753946 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:56:23.754909 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:23.754887 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 13:56:34.521492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.521461 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg"] Sep 16 13:56:34.521921 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.521662 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" podUID="bfa50451-5e01-4e4c-9a90-ed7a03939e09" containerName="maas-api" containerID="cri-o://2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33" gracePeriod=30 Sep 16 13:56:34.770626 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.770598 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:34.832287 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.832213 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa50451-5e01-4e4c-9a90-ed7a03939e09-openshift-service-ca\") pod \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " Sep 16 13:56:34.832287 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.832283 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4blln\" (UniqueName: \"kubernetes.io/projected/bfa50451-5e01-4e4c-9a90-ed7a03939e09-kube-api-access-4blln\") pod \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " Sep 16 13:56:34.832511 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.832371 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls\") pod \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " Sep 16 13:56:34.832511 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.832388 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-metrics-certs\") pod \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\" (UID: \"bfa50451-5e01-4e4c-9a90-ed7a03939e09\") " Sep 16 13:56:34.832630 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.832578 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa50451-5e01-4e4c-9a90-ed7a03939e09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bfa50451-5e01-4e4c-9a90-ed7a03939e09" (UID: "bfa50451-5e01-4e4c-9a90-ed7a03939e09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:34.832729 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.832712 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa50451-5e01-4e4c-9a90-ed7a03939e09-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:56:34.834520 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.834492 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bfa50451-5e01-4e4c-9a90-ed7a03939e09" (UID: "bfa50451-5e01-4e4c-9a90-ed7a03939e09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:34.834647 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.834519 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bfa50451-5e01-4e4c-9a90-ed7a03939e09" (UID: "bfa50451-5e01-4e4c-9a90-ed7a03939e09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:34.834647 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.834533 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa50451-5e01-4e4c-9a90-ed7a03939e09-kube-api-access-4blln" (OuterVolumeSpecName: "kube-api-access-4blln") pod "bfa50451-5e01-4e4c-9a90-ed7a03939e09" (UID: "bfa50451-5e01-4e4c-9a90-ed7a03939e09"). InnerVolumeSpecName "kube-api-access-4blln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:34.933933 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.933900 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:56:34.933933 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.933925 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa50451-5e01-4e4c-9a90-ed7a03939e09-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:56:34.933933 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:34.933934 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4blln\" (UniqueName: \"kubernetes.io/projected/bfa50451-5e01-4e4c-9a90-ed7a03939e09-kube-api-access-4blln\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:56:35.024325 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:35.024284 2581 generic.go:356] "Generic (PLEG): container finished" podID="bfa50451-5e01-4e4c-9a90-ed7a03939e09" containerID="2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33" exitCode=0 Sep 16 13:56:35.024517 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:35.024342 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" Sep 16 13:56:35.024517 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:35.024351 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" event={"ID":"bfa50451-5e01-4e4c-9a90-ed7a03939e09","Type":"ContainerDied","Data":"2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33"} Sep 16 13:56:35.024517 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:35.024404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg" event={"ID":"bfa50451-5e01-4e4c-9a90-ed7a03939e09","Type":"ContainerDied","Data":"d489ab03ddeb79cf4e04046198b89fcfee3409d2f70f2b579fe3ff7938b9f6b9"} Sep 16 13:56:35.024517 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:35.024435 2581 scope.go:117] "RemoveContainer" containerID="2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33" Sep 16 13:56:35.036561 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:35.036539 2581 scope.go:117] "RemoveContainer" containerID="2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33" Sep 16 13:56:35.036842 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:56:35.036821 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33\": container with ID starting with 2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33 not found: ID does not exist" containerID="2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33" Sep 16 13:56:35.036887 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:35.036853 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33"} err="failed to get container status \"2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33\": rpc error: code = NotFound desc = could not find container \"2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33\": container with ID starting with 2c517b0010982c25cf11d60dd9125f2f98791f36002e68a5f152604fa7b98b33 not found: ID does not exist" Sep 16 13:56:35.053487 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:35.053461 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg"] Sep 16 13:56:35.055952 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:35.055931 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1dcc64e3-86c8476cf9-c4rbg"] Sep 16 13:56:36.516969 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:36.516939 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfa50451-5e01-4e4c-9a90-ed7a03939e09" path="/var/lib/kubelet/pods/bfa50451-5e01-4e4c-9a90-ed7a03939e09/volumes" Sep 16 13:56:40.984161 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:40.984117 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22"] Sep 16 13:56:40.984746 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:40.984725 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa50451-5e01-4e4c-9a90-ed7a03939e09" containerName="maas-api" Sep 16 13:56:40.984835 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:40.984748 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa50451-5e01-4e4c-9a90-ed7a03939e09" containerName="maas-api" Sep 16 13:56:40.984906 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:40.984892 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa50451-5e01-4e4c-9a90-ed7a03939e09" containerName="maas-api" Sep 16 13:56:40.989675 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:40.989638 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:40.992171 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:40.992135 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ce782c2e-serving-cert\"" Sep 16 13:56:41.002200 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.002176 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22"] Sep 16 13:56:41.094318 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.094281 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-openshift-service-ca\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.094490 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.094331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6drv4\" (UniqueName: \"kubernetes.io/projected/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-kube-api-access-6drv4\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.094490 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.094359 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-maas-api-tls\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.094490 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.094412 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-metrics-certs\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.194999 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.194967 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6drv4\" (UniqueName: \"kubernetes.io/projected/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-kube-api-access-6drv4\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.195205 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.195013 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-maas-api-tls\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.195205 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.195068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-metrics-certs\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.195345 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.195225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-openshift-service-ca\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.195801 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.195777 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-openshift-service-ca\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.197511 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.197488 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-metrics-certs\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.197617 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.197545 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-maas-api-tls\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.205879 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.205852 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6drv4\" (UniqueName: \"kubernetes.io/projected/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-kube-api-access-6drv4\") pod \"maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.302862 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.302801 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:41.590510 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:41.590479 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22"] Sep 16 13:56:41.592389 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:56:41.592341 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e59e314_5e7e_4f4c_8c6a_8437c2867ecb.slice/crio-5c0cfcf1d15619a224aaef70e1a421aaa06f85bbcbe61d59095a7a2114e5cb7c WatchSource:0}: Error finding container 5c0cfcf1d15619a224aaef70e1a421aaa06f85bbcbe61d59095a7a2114e5cb7c: Status 404 returned error can't find the container with id 5c0cfcf1d15619a224aaef70e1a421aaa06f85bbcbe61d59095a7a2114e5cb7c Sep 16 13:56:42.067129 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:42.067104 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" event={"ID":"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb","Type":"ContainerStarted","Data":"5c0cfcf1d15619a224aaef70e1a421aaa06f85bbcbe61d59095a7a2114e5cb7c"} Sep 16 13:56:43.074550 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:43.074514 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" event={"ID":"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb","Type":"ContainerStarted","Data":"f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab"} Sep 16 13:56:43.074900 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:43.074644 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:56:43.100388 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:43.100338 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" podStartSLOduration=2.675096623 podStartE2EDuration="3.100324147s" totalImagesPullingTime="425.227524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:43.097239118 +0000 UTC m=+837.060597949" watchObservedRunningTime="2026-09-16 13:56:43.100324147 +0000 UTC m=+837.063682973" Sep 16 13:56:49.086569 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:56:49.086534 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:57:20.796236 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:20.796199 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22"] Sep 16 13:57:20.796777 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:20.796492 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" podUID="1e59e314-5e7e-4f4c-8c6a-8437c2867ecb" containerName="maas-api" containerID="cri-o://f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab" gracePeriod=30 Sep 16 13:57:21.036655 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.036629 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:57:21.186790 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.186691 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6drv4\" (UniqueName: \"kubernetes.io/projected/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-kube-api-access-6drv4\") pod \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " Sep 16 13:57:21.186949 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.186859 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-maas-api-tls\") pod \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " Sep 16 13:57:21.186949 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.186895 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-openshift-service-ca\") pod \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " Sep 16 13:57:21.186949 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.186924 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-metrics-certs\") pod \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\" (UID: \"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb\") " Sep 16 13:57:21.187494 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.187453 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e59e314-5e7e-4f4c-8c6a-8437c2867ecb" (UID: "1e59e314-5e7e-4f4c-8c6a-8437c2867ecb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:21.189490 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.189403 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e59e314-5e7e-4f4c-8c6a-8437c2867ecb" (UID: "1e59e314-5e7e-4f4c-8c6a-8437c2867ecb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:21.189856 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.189830 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-kube-api-access-6drv4" (OuterVolumeSpecName: "kube-api-access-6drv4") pod "1e59e314-5e7e-4f4c-8c6a-8437c2867ecb" (UID: "1e59e314-5e7e-4f4c-8c6a-8437c2867ecb"). InnerVolumeSpecName "kube-api-access-6drv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:21.190006 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.189978 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1e59e314-5e7e-4f4c-8c6a-8437c2867ecb" (UID: "1e59e314-5e7e-4f4c-8c6a-8437c2867ecb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:21.288176 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.288132 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:57:21.288176 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.288173 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:57:21.288176 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.288185 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:57:21.288481 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.288194 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6drv4\" (UniqueName: \"kubernetes.io/projected/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb-kube-api-access-6drv4\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:57:21.318090 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.315016 2581 generic.go:356] "Generic (PLEG): container finished" podID="1e59e314-5e7e-4f4c-8c6a-8437c2867ecb" containerID="f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab" exitCode=0 Sep 16 13:57:21.318090 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.315077 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" Sep 16 13:57:21.318090 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.315102 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" event={"ID":"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb","Type":"ContainerDied","Data":"f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab"} Sep 16 13:57:21.318090 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.315184 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22" event={"ID":"1e59e314-5e7e-4f4c-8c6a-8437c2867ecb","Type":"ContainerDied","Data":"5c0cfcf1d15619a224aaef70e1a421aaa06f85bbcbe61d59095a7a2114e5cb7c"} Sep 16 13:57:21.318090 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.315220 2581 scope.go:117] "RemoveContainer" containerID="f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab" Sep 16 13:57:21.330235 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.330201 2581 scope.go:117] "RemoveContainer" containerID="f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab" Sep 16 13:57:21.330619 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:57:21.330597 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab\": container with ID starting with f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab not found: ID does not exist" containerID="f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab" Sep 16 13:57:21.330697 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.330629 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab"} err="failed to get container status \"f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab\": rpc error: code = NotFound desc = could not find container \"f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab\": container with ID starting with f9a8fef4edfb362cc3281dae5a2e46fbfb479a5b93aec74918af2ed7513792ab not found: ID does not exist" Sep 16 13:57:21.343920 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.343889 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22"] Sep 16 13:57:21.346754 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:21.346728 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ce782c2e-5d49d49994-xnb22"] Sep 16 13:57:22.516150 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:22.516103 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e59e314-5e7e-4f4c-8c6a-8437c2867ecb" path="/var/lib/kubelet/pods/1e59e314-5e7e-4f4c-8c6a-8437c2867ecb/volumes" Sep 16 13:57:27.289028 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.288995 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6"] Sep 16 13:57:27.289658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.289558 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e59e314-5e7e-4f4c-8c6a-8437c2867ecb" containerName="maas-api" Sep 16 13:57:27.289658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.289573 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e59e314-5e7e-4f4c-8c6a-8437c2867ecb" containerName="maas-api" Sep 16 13:57:27.289658 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.289647 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e59e314-5e7e-4f4c-8c6a-8437c2867ecb" containerName="maas-api" Sep 16 13:57:27.294169 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.294129 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.297314 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.297298 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-38e8bf38-serving-cert\"" Sep 16 13:57:27.306279 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.306256 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6"] Sep 16 13:57:27.446018 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.445986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-maas-api-tls\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.446225 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.446114 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zqq\" (UniqueName: \"kubernetes.io/projected/e48d3f1c-e045-42ac-9f57-825861b203df-kube-api-access-f4zqq\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.446299 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.446245 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e48d3f1c-e045-42ac-9f57-825861b203df-openshift-service-ca\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.446363 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.446305 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-metrics-certs\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.547576 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.547491 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e48d3f1c-e045-42ac-9f57-825861b203df-openshift-service-ca\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.547576 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.547543 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-metrics-certs\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.547789 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.547580 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-maas-api-tls\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.547789 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.547661 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zqq\" (UniqueName: \"kubernetes.io/projected/e48d3f1c-e045-42ac-9f57-825861b203df-kube-api-access-f4zqq\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.547901 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:57:27.547794 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-serving-cert: secret "maas-api-e2e-ait-38e8bf38-serving-cert" not found Sep 16 13:57:27.547901 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:57:27.547877 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-maas-api-tls podName:e48d3f1c-e045-42ac-9f57-825861b203df nodeName:}" failed. No retries permitted until 2026-09-16 13:57:28.047856823 +0000 UTC m=+882.011215629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-maas-api-tls") pod "maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" (UID: "e48d3f1c-e045-42ac-9f57-825861b203df") : secret "maas-api-e2e-ait-38e8bf38-serving-cert" not found Sep 16 13:57:27.548287 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.548268 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e48d3f1c-e045-42ac-9f57-825861b203df-openshift-service-ca\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.550037 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.550010 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-metrics-certs\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:27.557156 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:27.557121 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zqq\" (UniqueName: \"kubernetes.io/projected/e48d3f1c-e045-42ac-9f57-825861b203df-kube-api-access-f4zqq\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:28.051733 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:28.051699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-maas-api-tls\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:28.054222 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:28.054198 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-maas-api-tls\") pod \"maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:28.205984 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:28.205947 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:28.493025 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:28.493000 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6"] Sep 16 13:57:28.495174 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:57:28.495134 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48d3f1c_e045_42ac_9f57_825861b203df.slice/crio-11e3bd2aa592b96b41a55ac9b84da715d92c332b947f2a8a2365a4b394450e61 WatchSource:0}: Error finding container 11e3bd2aa592b96b41a55ac9b84da715d92c332b947f2a8a2365a4b394450e61: Status 404 returned error can't find the container with id 11e3bd2aa592b96b41a55ac9b84da715d92c332b947f2a8a2365a4b394450e61 Sep 16 13:57:29.366868 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:29.366830 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" event={"ID":"e48d3f1c-e045-42ac-9f57-825861b203df","Type":"ContainerStarted","Data":"a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860"} Sep 16 13:57:29.367057 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:29.366907 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" event={"ID":"e48d3f1c-e045-42ac-9f57-825861b203df","Type":"ContainerStarted","Data":"11e3bd2aa592b96b41a55ac9b84da715d92c332b947f2a8a2365a4b394450e61"} Sep 16 13:57:29.367057 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:29.366972 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:29.399891 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:29.399846 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" podStartSLOduration=2.054711735 podStartE2EDuration="2.399832577s" totalImagesPullingTime="345.120842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:29.396210428 +0000 UTC m=+883.359569278" watchObservedRunningTime="2026-09-16 13:57:29.399832577 +0000 UTC m=+883.363191444" Sep 16 13:57:35.378588 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:35.378554 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:57:46.547619 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:46.547536 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 13:57:46.548015 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:46.548002 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 13:57:46.557106 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:46.557085 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:57:46.557661 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:57:46.557645 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 13:58:11.991224 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:11.991188 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6"] Sep 16 13:58:11.991762 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:11.991413 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" podUID="e48d3f1c-e045-42ac-9f57-825861b203df" containerName="maas-api" containerID="cri-o://a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860" gracePeriod=30 Sep 16 13:58:12.250782 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.250720 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:58:12.368388 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.368352 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e48d3f1c-e045-42ac-9f57-825861b203df-openshift-service-ca\") pod \"e48d3f1c-e045-42ac-9f57-825861b203df\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " Sep 16 13:58:12.368592 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.368441 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4zqq\" (UniqueName: \"kubernetes.io/projected/e48d3f1c-e045-42ac-9f57-825861b203df-kube-api-access-f4zqq\") pod \"e48d3f1c-e045-42ac-9f57-825861b203df\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " Sep 16 13:58:12.368592 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.368465 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-metrics-certs\") pod \"e48d3f1c-e045-42ac-9f57-825861b203df\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " Sep 16 13:58:12.368592 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.368532 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-maas-api-tls\") pod \"e48d3f1c-e045-42ac-9f57-825861b203df\" (UID: \"e48d3f1c-e045-42ac-9f57-825861b203df\") " Sep 16 13:58:12.368780 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.368757 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48d3f1c-e045-42ac-9f57-825861b203df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e48d3f1c-e045-42ac-9f57-825861b203df" (UID: "e48d3f1c-e045-42ac-9f57-825861b203df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:58:12.370699 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.370666 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48d3f1c-e045-42ac-9f57-825861b203df-kube-api-access-f4zqq" (OuterVolumeSpecName: "kube-api-access-f4zqq") pod "e48d3f1c-e045-42ac-9f57-825861b203df" (UID: "e48d3f1c-e045-42ac-9f57-825861b203df"). InnerVolumeSpecName "kube-api-access-f4zqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:12.370699 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.370685 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e48d3f1c-e045-42ac-9f57-825861b203df" (UID: "e48d3f1c-e045-42ac-9f57-825861b203df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:58:12.370887 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.370720 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e48d3f1c-e045-42ac-9f57-825861b203df" (UID: "e48d3f1c-e045-42ac-9f57-825861b203df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:58:12.470027 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.469994 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:58:12.470027 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.470026 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e48d3f1c-e045-42ac-9f57-825861b203df-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:58:12.470027 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.470036 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4zqq\" (UniqueName: \"kubernetes.io/projected/e48d3f1c-e045-42ac-9f57-825861b203df-kube-api-access-f4zqq\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:58:12.470271 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.470045 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e48d3f1c-e045-42ac-9f57-825861b203df-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:58:12.640613 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.640578 2581 generic.go:356] "Generic (PLEG): container finished" podID="e48d3f1c-e045-42ac-9f57-825861b203df" containerID="a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860" exitCode=0 Sep 16 13:58:12.640791 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.640625 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" Sep 16 13:58:12.640791 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.640666 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" event={"ID":"e48d3f1c-e045-42ac-9f57-825861b203df","Type":"ContainerDied","Data":"a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860"} Sep 16 13:58:12.640791 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.640722 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6" event={"ID":"e48d3f1c-e045-42ac-9f57-825861b203df","Type":"ContainerDied","Data":"11e3bd2aa592b96b41a55ac9b84da715d92c332b947f2a8a2365a4b394450e61"} Sep 16 13:58:12.640791 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.640753 2581 scope.go:117] "RemoveContainer" containerID="a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860" Sep 16 13:58:12.650448 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.650431 2581 scope.go:117] "RemoveContainer" containerID="a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860" Sep 16 13:58:12.650730 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:58:12.650714 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860\": container with ID starting with a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860 not found: ID does not exist" containerID="a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860" Sep 16 13:58:12.650785 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.650737 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860"} err="failed to get container status \"a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860\": rpc error: code = NotFound desc = could not find container \"a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860\": container with ID starting with a5f6bf03a0427ef1eb7a44ca8ee81bbc9b2816f4fa0f32d4cbf44ba48d74e860 not found: ID does not exist" Sep 16 13:58:12.675227 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.675208 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6"] Sep 16 13:58:12.685404 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:12.685379 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38e8bf38-6f777b774c-xk9r6"] Sep 16 13:58:14.523842 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:14.518803 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e48d3f1c-e045-42ac-9f57-825861b203df" path="/var/lib/kubelet/pods/e48d3f1c-e045-42ac-9f57-825861b203df/volumes" Sep 16 13:58:17.537346 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.537307 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj"] Sep 16 13:58:17.537789 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.537554 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" podUID="25215817-6cd2-4ea9-a239-c7c7a7d5bf10" containerName="maas-api" containerID="cri-o://5279c6954c3683f6b6f06a9294c11183a7a2922b2fb5634e5cf0ad0eafba3322" gracePeriod=30 Sep 16 13:58:17.676782 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.676743 2581 generic.go:356] "Generic (PLEG): container finished" podID="25215817-6cd2-4ea9-a239-c7c7a7d5bf10" containerID="5279c6954c3683f6b6f06a9294c11183a7a2922b2fb5634e5cf0ad0eafba3322" exitCode=0 Sep 16 13:58:17.676971 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.676815 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" event={"ID":"25215817-6cd2-4ea9-a239-c7c7a7d5bf10","Type":"ContainerDied","Data":"5279c6954c3683f6b6f06a9294c11183a7a2922b2fb5634e5cf0ad0eafba3322"} Sep 16 13:58:17.802527 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.802452 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:58:17.818000 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.817972 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbpmb\" (UniqueName: \"kubernetes.io/projected/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-kube-api-access-jbpmb\") pod \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " Sep 16 13:58:17.818259 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.818013 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-maas-api-tls\") pod \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " Sep 16 13:58:17.818259 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.818090 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-metrics-certs\") pod \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " Sep 16 13:58:17.818259 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.818230 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-openshift-service-ca\") pod \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\" (UID: \"25215817-6cd2-4ea9-a239-c7c7a7d5bf10\") " Sep 16 13:58:17.818854 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.818825 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25215817-6cd2-4ea9-a239-c7c7a7d5bf10" (UID: "25215817-6cd2-4ea9-a239-c7c7a7d5bf10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:58:17.820529 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.820498 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-kube-api-access-jbpmb" (OuterVolumeSpecName: "kube-api-access-jbpmb") pod "25215817-6cd2-4ea9-a239-c7c7a7d5bf10" (UID: "25215817-6cd2-4ea9-a239-c7c7a7d5bf10"). InnerVolumeSpecName "kube-api-access-jbpmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:17.820642 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.820552 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25215817-6cd2-4ea9-a239-c7c7a7d5bf10" (UID: "25215817-6cd2-4ea9-a239-c7c7a7d5bf10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:58:17.820642 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.820587 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "25215817-6cd2-4ea9-a239-c7c7a7d5bf10" (UID: "25215817-6cd2-4ea9-a239-c7c7a7d5bf10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:58:17.919934 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.919901 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:58:17.919934 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.919933 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:58:17.920181 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.919948 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbpmb\" (UniqueName: \"kubernetes.io/projected/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-kube-api-access-jbpmb\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:58:17.920181 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:17.919959 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25215817-6cd2-4ea9-a239-c7c7a7d5bf10-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:58:18.683941 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:18.683914 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" Sep 16 13:58:18.684367 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:18.683954 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj" event={"ID":"25215817-6cd2-4ea9-a239-c7c7a7d5bf10","Type":"ContainerDied","Data":"31fe0ee3d506761cefd62ca45ac1a9dea1c148a49973930140445c98d09cb6f0"} Sep 16 13:58:18.684367 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:18.684014 2581 scope.go:117] "RemoveContainer" containerID="5279c6954c3683f6b6f06a9294c11183a7a2922b2fb5634e5cf0ad0eafba3322" Sep 16 13:58:18.708419 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:18.708387 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj"] Sep 16 13:58:18.714262 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:18.714238 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-617597-5b6877bf86-nxbnj"] Sep 16 13:58:20.516027 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:20.515990 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25215817-6cd2-4ea9-a239-c7c7a7d5bf10" path="/var/lib/kubelet/pods/25215817-6cd2-4ea9-a239-c7c7a7d5bf10/volumes" Sep 16 13:58:22.267065 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.267031 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d"] Sep 16 13:58:22.267629 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.267613 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25215817-6cd2-4ea9-a239-c7c7a7d5bf10" containerName="maas-api" Sep 16 13:58:22.267681 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.267632 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25215817-6cd2-4ea9-a239-c7c7a7d5bf10" containerName="maas-api" Sep 16 13:58:22.267681 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.267653 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48d3f1c-e045-42ac-9f57-825861b203df" containerName="maas-api" Sep 16 13:58:22.267681 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.267659 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48d3f1c-e045-42ac-9f57-825861b203df" containerName="maas-api" Sep 16 13:58:22.267783 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.267724 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48d3f1c-e045-42ac-9f57-825861b203df" containerName="maas-api" Sep 16 13:58:22.267783 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.267732 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25215817-6cd2-4ea9-a239-c7c7a7d5bf10" containerName="maas-api" Sep 16 13:58:22.272331 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.272312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.275209 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.275184 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1852eb66-serving-cert\"" Sep 16 13:58:22.288311 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.288286 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d"] Sep 16 13:58:22.363656 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.363624 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29z2\" (UniqueName: \"kubernetes.io/projected/2b115f88-01a0-42f8-a27a-8112233d7a54-kube-api-access-m29z2\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.363785 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.363662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-metrics-certs\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.363785 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.363692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b115f88-01a0-42f8-a27a-8112233d7a54-openshift-service-ca\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.363785 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.363749 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-maas-api-tls\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.465079 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.465043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29z2\" (UniqueName: \"kubernetes.io/projected/2b115f88-01a0-42f8-a27a-8112233d7a54-kube-api-access-m29z2\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.465290 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.465089 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-metrics-certs\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.465290 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.465137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b115f88-01a0-42f8-a27a-8112233d7a54-openshift-service-ca\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.465290 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.465219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-maas-api-tls\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.465964 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.465934 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b115f88-01a0-42f8-a27a-8112233d7a54-openshift-service-ca\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.467607 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.467584 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-maas-api-tls\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.467722 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.467615 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-metrics-certs\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.474962 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.474940 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29z2\" (UniqueName: \"kubernetes.io/projected/2b115f88-01a0-42f8-a27a-8112233d7a54-kube-api-access-m29z2\") pod \"maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.582950 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.582883 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:22.859720 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:22.859690 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d"] Sep 16 13:58:22.860982 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:58:22.860956 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b115f88_01a0_42f8_a27a_8112233d7a54.slice/crio-788994a37254315b3c3aba54c4a130ada6dbefdf5d0accccf91d3594c9ce92d0 WatchSource:0}: Error finding container 788994a37254315b3c3aba54c4a130ada6dbefdf5d0accccf91d3594c9ce92d0: Status 404 returned error can't find the container with id 788994a37254315b3c3aba54c4a130ada6dbefdf5d0accccf91d3594c9ce92d0 Sep 16 13:58:23.717338 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:23.717302 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" event={"ID":"2b115f88-01a0-42f8-a27a-8112233d7a54","Type":"ContainerStarted","Data":"7c8d62d196681664f62f6ded950ba3e94b9a56971761345e52f98696f9c61c6f"} Sep 16 13:58:23.717692 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:23.717359 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" event={"ID":"2b115f88-01a0-42f8-a27a-8112233d7a54","Type":"ContainerStarted","Data":"788994a37254315b3c3aba54c4a130ada6dbefdf5d0accccf91d3594c9ce92d0"} Sep 16 13:58:23.717692 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:23.717543 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:23.742746 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:23.742700 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" podStartSLOduration=1.3742841270000001 podStartE2EDuration="1.742687305s" totalImagesPullingTime="368.403178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:58:23.739281187 +0000 UTC m=+937.702640028" watchObservedRunningTime="2026-09-16 13:58:23.742687305 +0000 UTC m=+937.706046131" Sep 16 13:58:29.730004 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:29.729975 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:58:59.681075 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:59.681044 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d"] Sep 16 13:58:59.681538 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:59.681293 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" podUID="2b115f88-01a0-42f8-a27a-8112233d7a54" containerName="maas-api" containerID="cri-o://7c8d62d196681664f62f6ded950ba3e94b9a56971761345e52f98696f9c61c6f" gracePeriod=30 Sep 16 13:58:59.725583 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:59.725542 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" podUID="2b115f88-01a0-42f8-a27a-8112233d7a54" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.66:8443/health\": dial tcp 10.132.0.66:8443: connect: connection refused" Sep 16 13:58:59.939325 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:59.939270 2581 generic.go:356] "Generic (PLEG): container finished" podID="2b115f88-01a0-42f8-a27a-8112233d7a54" containerID="7c8d62d196681664f62f6ded950ba3e94b9a56971761345e52f98696f9c61c6f" exitCode=0 Sep 16 13:58:59.939424 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:59.939340 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" event={"ID":"2b115f88-01a0-42f8-a27a-8112233d7a54","Type":"ContainerDied","Data":"7c8d62d196681664f62f6ded950ba3e94b9a56971761345e52f98696f9c61c6f"} Sep 16 13:58:59.939424 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:59.939393 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" event={"ID":"2b115f88-01a0-42f8-a27a-8112233d7a54","Type":"ContainerDied","Data":"788994a37254315b3c3aba54c4a130ada6dbefdf5d0accccf91d3594c9ce92d0"} Sep 16 13:58:59.939424 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:59.939412 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="788994a37254315b3c3aba54c4a130ada6dbefdf5d0accccf91d3594c9ce92d0" Sep 16 13:58:59.941305 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:58:59.941278 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:59:00.107395 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.107362 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-maas-api-tls\") pod \"2b115f88-01a0-42f8-a27a-8112233d7a54\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " Sep 16 13:59:00.107592 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.107441 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-metrics-certs\") pod \"2b115f88-01a0-42f8-a27a-8112233d7a54\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " Sep 16 13:59:00.107592 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.107501 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b115f88-01a0-42f8-a27a-8112233d7a54-openshift-service-ca\") pod \"2b115f88-01a0-42f8-a27a-8112233d7a54\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " Sep 16 13:59:00.107592 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.107534 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m29z2\" (UniqueName: \"kubernetes.io/projected/2b115f88-01a0-42f8-a27a-8112233d7a54-kube-api-access-m29z2\") pod \"2b115f88-01a0-42f8-a27a-8112233d7a54\" (UID: \"2b115f88-01a0-42f8-a27a-8112233d7a54\") " Sep 16 13:59:00.107918 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.107863 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b115f88-01a0-42f8-a27a-8112233d7a54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b115f88-01a0-42f8-a27a-8112233d7a54" (UID: "2b115f88-01a0-42f8-a27a-8112233d7a54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:59:00.108117 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.108094 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b115f88-01a0-42f8-a27a-8112233d7a54-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:59:00.110201 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.110176 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b115f88-01a0-42f8-a27a-8112233d7a54" (UID: "2b115f88-01a0-42f8-a27a-8112233d7a54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:59:00.110336 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.110183 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b115f88-01a0-42f8-a27a-8112233d7a54-kube-api-access-m29z2" (OuterVolumeSpecName: "kube-api-access-m29z2") pod "2b115f88-01a0-42f8-a27a-8112233d7a54" (UID: "2b115f88-01a0-42f8-a27a-8112233d7a54"). InnerVolumeSpecName "kube-api-access-m29z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:59:00.110336 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.110238 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2b115f88-01a0-42f8-a27a-8112233d7a54" (UID: "2b115f88-01a0-42f8-a27a-8112233d7a54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:59:00.209060 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.209038 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:59:00.209221 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.209062 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b115f88-01a0-42f8-a27a-8112233d7a54-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:59:00.209221 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.209072 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m29z2\" (UniqueName: \"kubernetes.io/projected/2b115f88-01a0-42f8-a27a-8112233d7a54-kube-api-access-m29z2\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 13:59:00.945941 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.945862 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d" Sep 16 13:59:00.978607 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.978577 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d"] Sep 16 13:59:00.983310 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:00.983285 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1852eb66-7bffcc4596-bqq5d"] Sep 16 13:59:02.522801 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:02.522769 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b115f88-01a0-42f8-a27a-8112233d7a54" path="/var/lib/kubelet/pods/2b115f88-01a0-42f8-a27a-8112233d7a54/volumes" Sep 16 13:59:12.874158 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.874061 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g"] Sep 16 13:59:12.874510 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.874496 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b115f88-01a0-42f8-a27a-8112233d7a54" containerName="maas-api" Sep 16 13:59:12.874510 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.874507 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b115f88-01a0-42f8-a27a-8112233d7a54" containerName="maas-api" Sep 16 13:59:12.874594 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.874583 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b115f88-01a0-42f8-a27a-8112233d7a54" containerName="maas-api" Sep 16 13:59:12.879047 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.879019 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:12.884976 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.884953 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-733e5b-serving-cert\"" Sep 16 13:59:12.914527 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.914500 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g"] Sep 16 13:59:12.929839 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.929813 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk79p\" (UniqueName: \"kubernetes.io/projected/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-kube-api-access-qk79p\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:12.929990 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.929858 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-maas-api-tls\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:12.929990 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.929964 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-metrics-certs\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:12.930128 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:12.929994 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-openshift-service-ca\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:13.030765 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:13.030728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-metrics-certs\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:13.030765 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:13.030766 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-openshift-service-ca\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:13.031002 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:13.030870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk79p\" (UniqueName: \"kubernetes.io/projected/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-kube-api-access-qk79p\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:13.031002 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:13.030929 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-maas-api-tls\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:13.031127 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:59:13.031103 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-serving-cert: secret "maas-api-e2e-explicit-733e5b-serving-cert" not found Sep 16 13:59:13.031237 ip-10-0-143-222 kubenswrapper[2581]: E0916 13:59:13.031217 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-maas-api-tls podName:8d54a84f-0aa2-454d-b2d4-115aad2d8a4e nodeName:}" failed. No retries permitted until 2026-09-16 13:59:13.531192068 +0000 UTC m=+987.494550876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-maas-api-tls") pod "maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" (UID: "8d54a84f-0aa2-454d-b2d4-115aad2d8a4e") : secret "maas-api-e2e-explicit-733e5b-serving-cert" not found Sep 16 13:59:13.031650 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:13.031627 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-openshift-service-ca\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:13.033292 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:13.033272 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-metrics-certs\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:13.043492 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:13.043467 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk79p\" (UniqueName: \"kubernetes.io/projected/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-kube-api-access-qk79p\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:13.536427 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:13.536392 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-maas-api-tls\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:13.539180 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:13.539127 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-maas-api-tls\") pod \"maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:13.790040 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:13.789967 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:14.077193 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:14.077132 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g"] Sep 16 13:59:14.078851 ip-10-0-143-222 kubenswrapper[2581]: W0916 13:59:14.078814 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d54a84f_0aa2_454d_b2d4_115aad2d8a4e.slice/crio-c9c4e423fa58e869b3ae7dcae6cb323303aa54ba43e87647829b8d4f5135594b WatchSource:0}: Error finding container c9c4e423fa58e869b3ae7dcae6cb323303aa54ba43e87647829b8d4f5135594b: Status 404 returned error can't find the container with id c9c4e423fa58e869b3ae7dcae6cb323303aa54ba43e87647829b8d4f5135594b Sep 16 13:59:15.030441 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:15.030407 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" event={"ID":"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e","Type":"ContainerStarted","Data":"4927a098cc1a0d815407515c113f16f0b3e060aa22d9938d741fe1a7d9b0f820"} Sep 16 13:59:15.030730 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:15.030473 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" event={"ID":"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e","Type":"ContainerStarted","Data":"c9c4e423fa58e869b3ae7dcae6cb323303aa54ba43e87647829b8d4f5135594b"} Sep 16 13:59:15.030730 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:15.030503 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 13:59:15.058130 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:15.058087 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" podStartSLOduration=2.20683375 podStartE2EDuration="3.05807182s" totalImagesPullingTime="851.23807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:15.055109865 +0000 UTC m=+989.018468693" watchObservedRunningTime="2026-09-16 13:59:15.05807182 +0000 UTC m=+989.021430647" Sep 16 13:59:21.041560 ip-10-0-143-222 kubenswrapper[2581]: I0916 13:59:21.041530 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 14:00:26.238078 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.238043 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g"] Sep 16 14:00:26.238680 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.238332 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" podUID="8d54a84f-0aa2-454d-b2d4-115aad2d8a4e" containerName="maas-api" containerID="cri-o://4927a098cc1a0d815407515c113f16f0b3e060aa22d9938d741fe1a7d9b0f820" gracePeriod=30 Sep 16 14:00:26.487843 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.487811 2581 generic.go:356] "Generic (PLEG): container finished" podID="8d54a84f-0aa2-454d-b2d4-115aad2d8a4e" containerID="4927a098cc1a0d815407515c113f16f0b3e060aa22d9938d741fe1a7d9b0f820" exitCode=0 Sep 16 14:00:26.487981 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.487894 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" event={"ID":"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e","Type":"ContainerDied","Data":"4927a098cc1a0d815407515c113f16f0b3e060aa22d9938d741fe1a7d9b0f820"} Sep 16 14:00:26.487981 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.487951 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" event={"ID":"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e","Type":"ContainerDied","Data":"c9c4e423fa58e869b3ae7dcae6cb323303aa54ba43e87647829b8d4f5135594b"} Sep 16 14:00:26.487981 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.487974 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9c4e423fa58e869b3ae7dcae6cb323303aa54ba43e87647829b8d4f5135594b" Sep 16 14:00:26.503443 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.503381 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 14:00:26.622159 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.622101 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-openshift-service-ca\") pod \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " Sep 16 14:00:26.622438 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.622171 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk79p\" (UniqueName: \"kubernetes.io/projected/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-kube-api-access-qk79p\") pod \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " Sep 16 14:00:26.622438 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.622259 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-maas-api-tls\") pod \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " Sep 16 14:00:26.622438 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.622318 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-metrics-certs\") pod \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\" (UID: \"8d54a84f-0aa2-454d-b2d4-115aad2d8a4e\") " Sep 16 14:00:26.622646 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.622547 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d54a84f-0aa2-454d-b2d4-115aad2d8a4e" (UID: "8d54a84f-0aa2-454d-b2d4-115aad2d8a4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:00:26.622811 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.622789 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:00:26.625587 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.625552 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8d54a84f-0aa2-454d-b2d4-115aad2d8a4e" (UID: "8d54a84f-0aa2-454d-b2d4-115aad2d8a4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:00:26.625811 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.625787 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d54a84f-0aa2-454d-b2d4-115aad2d8a4e" (UID: "8d54a84f-0aa2-454d-b2d4-115aad2d8a4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:00:26.625924 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.625908 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-kube-api-access-qk79p" (OuterVolumeSpecName: "kube-api-access-qk79p") pod "8d54a84f-0aa2-454d-b2d4-115aad2d8a4e" (UID: "8d54a84f-0aa2-454d-b2d4-115aad2d8a4e"). InnerVolumeSpecName "kube-api-access-qk79p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:00:26.723489 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.723458 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk79p\" (UniqueName: \"kubernetes.io/projected/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-kube-api-access-qk79p\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:00:26.723489 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.723485 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:00:26.723489 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:26.723496 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:00:27.493509 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:27.493472 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g" Sep 16 14:00:27.519993 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:27.519964 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g"] Sep 16 14:00:27.527296 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:27.527269 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-733e5b-55d74f8867-wvh6g"] Sep 16 14:00:28.515897 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:28.515865 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d54a84f-0aa2-454d-b2d4-115aad2d8a4e" path="/var/lib/kubelet/pods/8d54a84f-0aa2-454d-b2d4-115aad2d8a4e/volumes" Sep 16 14:00:48.783330 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.783249 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk"] Sep 16 14:00:48.783929 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.783907 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d54a84f-0aa2-454d-b2d4-115aad2d8a4e" containerName="maas-api" Sep 16 14:00:48.784025 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.783931 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d54a84f-0aa2-454d-b2d4-115aad2d8a4e" containerName="maas-api" Sep 16 14:00:48.784084 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.784053 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d54a84f-0aa2-454d-b2d4-115aad2d8a4e" containerName="maas-api" Sep 16 14:00:48.787522 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.787500 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:48.789906 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.789885 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2b5aadeb-serving-cert\"" Sep 16 14:00:48.796891 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.796498 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk"] Sep 16 14:00:48.918274 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.918239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:48.918441 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.918291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-metrics-certs\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:48.918441 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.918310 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85lt\" (UniqueName: \"kubernetes.io/projected/59d40f29-0fcc-4ae7-924e-add20e7f7be8-kube-api-access-q85lt\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:48.918441 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:48.918343 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d40f29-0fcc-4ae7-924e-add20e7f7be8-openshift-service-ca\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:49.019815 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:49.019782 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d40f29-0fcc-4ae7-924e-add20e7f7be8-openshift-service-ca\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:49.019971 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:49.019881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:49.019971 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:49.019916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-metrics-certs\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:49.019971 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:49.019933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85lt\" (UniqueName: \"kubernetes.io/projected/59d40f29-0fcc-4ae7-924e-add20e7f7be8-kube-api-access-q85lt\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:49.020154 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:00:49.020030 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-serving-cert: secret "maas-api-e2e-mt-2b5aadeb-serving-cert" not found Sep 16 14:00:49.020154 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:00:49.020120 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls podName:59d40f29-0fcc-4ae7-924e-add20e7f7be8 nodeName:}" failed. No retries permitted until 2026-09-16 14:00:49.520097635 +0000 UTC m=+1083.483456455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls") pod "maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" (UID: "59d40f29-0fcc-4ae7-924e-add20e7f7be8") : secret "maas-api-e2e-mt-2b5aadeb-serving-cert" not found Sep 16 14:00:49.020571 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:49.020549 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d40f29-0fcc-4ae7-924e-add20e7f7be8-openshift-service-ca\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:49.022431 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:49.022414 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-metrics-certs\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:49.032271 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:49.032247 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85lt\" (UniqueName: \"kubernetes.io/projected/59d40f29-0fcc-4ae7-924e-add20e7f7be8-kube-api-access-q85lt\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:49.524867 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:49.524826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:49.525048 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:00:49.524973 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-serving-cert: secret "maas-api-e2e-mt-2b5aadeb-serving-cert" not found Sep 16 14:00:49.525048 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:00:49.525043 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls podName:59d40f29-0fcc-4ae7-924e-add20e7f7be8 nodeName:}" failed. No retries permitted until 2026-09-16 14:00:50.525027325 +0000 UTC m=+1084.488386130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls") pod "maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" (UID: "59d40f29-0fcc-4ae7-924e-add20e7f7be8") : secret "maas-api-e2e-mt-2b5aadeb-serving-cert" not found Sep 16 14:00:50.534558 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:50.534516 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:50.536843 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:50.536817 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls\") pod \"maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:50.601569 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:50.601536 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:50.880356 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:50.880330 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk"] Sep 16 14:00:50.881780 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:00:50.881739 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d40f29_0fcc_4ae7_924e_add20e7f7be8.slice/crio-2b64c9e22135d86249406947947ca47f48be6e5f1b2ebd5f105f762ed9dd3e0c WatchSource:0}: Error finding container 2b64c9e22135d86249406947947ca47f48be6e5f1b2ebd5f105f762ed9dd3e0c: Status 404 returned error can't find the container with id 2b64c9e22135d86249406947947ca47f48be6e5f1b2ebd5f105f762ed9dd3e0c Sep 16 14:00:50.883317 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:50.883297 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:00:51.638868 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:51.638832 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" event={"ID":"59d40f29-0fcc-4ae7-924e-add20e7f7be8","Type":"ContainerStarted","Data":"2b64c9e22135d86249406947947ca47f48be6e5f1b2ebd5f105f762ed9dd3e0c"} Sep 16 14:00:52.646307 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:52.646270 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" event={"ID":"59d40f29-0fcc-4ae7-924e-add20e7f7be8","Type":"ContainerStarted","Data":"00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413"} Sep 16 14:00:52.646677 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:52.646387 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:00:52.666111 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:52.666062 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" podStartSLOduration=3.470030842 podStartE2EDuration="4.666046903s" totalImagesPullingTime="1.196016061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:52.66541364 +0000 UTC m=+1086.628772467" watchObservedRunningTime="2026-09-16 14:00:52.666046903 +0000 UTC m=+1086.629405730" Sep 16 14:00:58.659262 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:00:58.659221 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:01:33.119813 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.119780 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk"] Sep 16 14:01:33.122024 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.119993 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" podUID="59d40f29-0fcc-4ae7-924e-add20e7f7be8" containerName="maas-api" containerID="cri-o://00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413" gracePeriod=30 Sep 16 14:01:33.383259 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.383231 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:01:33.418078 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.418050 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-metrics-certs\") pod \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " Sep 16 14:01:33.418263 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.418102 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q85lt\" (UniqueName: \"kubernetes.io/projected/59d40f29-0fcc-4ae7-924e-add20e7f7be8-kube-api-access-q85lt\") pod \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " Sep 16 14:01:33.418263 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.418158 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls\") pod \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " Sep 16 14:01:33.418263 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.418191 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d40f29-0fcc-4ae7-924e-add20e7f7be8-openshift-service-ca\") pod \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\" (UID: \"59d40f29-0fcc-4ae7-924e-add20e7f7be8\") " Sep 16 14:01:33.418597 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.418568 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d40f29-0fcc-4ae7-924e-add20e7f7be8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59d40f29-0fcc-4ae7-924e-add20e7f7be8" (UID: "59d40f29-0fcc-4ae7-924e-add20e7f7be8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:01:33.420582 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.420544 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59d40f29-0fcc-4ae7-924e-add20e7f7be8" (UID: "59d40f29-0fcc-4ae7-924e-add20e7f7be8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:01:33.420582 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.420559 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d40f29-0fcc-4ae7-924e-add20e7f7be8-kube-api-access-q85lt" (OuterVolumeSpecName: "kube-api-access-q85lt") pod "59d40f29-0fcc-4ae7-924e-add20e7f7be8" (UID: "59d40f29-0fcc-4ae7-924e-add20e7f7be8"). InnerVolumeSpecName "kube-api-access-q85lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:01:33.420766 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.420740 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "59d40f29-0fcc-4ae7-924e-add20e7f7be8" (UID: "59d40f29-0fcc-4ae7-924e-add20e7f7be8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:01:33.519452 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.519419 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q85lt\" (UniqueName: \"kubernetes.io/projected/59d40f29-0fcc-4ae7-924e-add20e7f7be8-kube-api-access-q85lt\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:01:33.519452 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.519450 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:01:33.519452 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.519460 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d40f29-0fcc-4ae7-924e-add20e7f7be8-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:01:33.519749 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.519469 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d40f29-0fcc-4ae7-924e-add20e7f7be8-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:01:33.904325 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.904291 2581 generic.go:356] "Generic (PLEG): container finished" podID="59d40f29-0fcc-4ae7-924e-add20e7f7be8" containerID="00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413" exitCode=0 Sep 16 14:01:33.904325 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.904329 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" Sep 16 14:01:33.904545 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.904368 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" event={"ID":"59d40f29-0fcc-4ae7-924e-add20e7f7be8","Type":"ContainerDied","Data":"00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413"} Sep 16 14:01:33.904545 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.904428 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk" event={"ID":"59d40f29-0fcc-4ae7-924e-add20e7f7be8","Type":"ContainerDied","Data":"2b64c9e22135d86249406947947ca47f48be6e5f1b2ebd5f105f762ed9dd3e0c"} Sep 16 14:01:33.904545 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.904457 2581 scope.go:117] "RemoveContainer" containerID="00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413" Sep 16 14:01:33.914752 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.914731 2581 scope.go:117] "RemoveContainer" containerID="00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413" Sep 16 14:01:33.915031 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:01:33.915012 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413\": container with ID starting with 00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413 not found: ID does not exist" containerID="00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413" Sep 16 14:01:33.915076 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.915038 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413"} err="failed to get container status \"00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413\": rpc error: code = NotFound desc = could not find container \"00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413\": container with ID starting with 00d5db23b882f3bffd4622cbf3a60b5540713a3dd372aa777ee6857d3ae51413 not found: ID does not exist" Sep 16 14:01:33.929978 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.929955 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk"] Sep 16 14:01:33.935322 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:33.935299 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5aadeb-75686f5ff7-8zxmk"] Sep 16 14:01:34.517545 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:01:34.517507 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59d40f29-0fcc-4ae7-924e-add20e7f7be8" path="/var/lib/kubelet/pods/59d40f29-0fcc-4ae7-924e-add20e7f7be8/volumes" Sep 16 14:02:03.868793 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:03.868761 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz"] Sep 16 14:02:03.869382 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:03.869345 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d40f29-0fcc-4ae7-924e-add20e7f7be8" containerName="maas-api" Sep 16 14:02:03.869382 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:03.869365 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d40f29-0fcc-4ae7-924e-add20e7f7be8" containerName="maas-api" Sep 16 14:02:03.869501 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:03.869476 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d40f29-0fcc-4ae7-924e-add20e7f7be8" containerName="maas-api" Sep 16 14:02:03.879418 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:03.879391 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:03.882202 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:03.882178 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-740a25-serving-cert\"" Sep 16 14:02:03.892709 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:03.892685 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz"] Sep 16 14:02:04.003012 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.002976 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.003193 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.003019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.003193 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.003115 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.003193 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.003187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlvfp\" (UniqueName: \"kubernetes.io/projected/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-kube-api-access-hlvfp\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.104732 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.104692 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.104910 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.104787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlvfp\" (UniqueName: \"kubernetes.io/projected/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-kube-api-access-hlvfp\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.104910 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.104872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.104910 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.104905 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.105802 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.105768 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.106154 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:02:04.106115 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-serving-cert: secret "maas-api-e2e-shared-a-w2-740a25-serving-cert" not found Sep 16 14:02:04.106277 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:02:04.106228 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-maas-api-tls podName:3748a3db-5c2d-4303-85e3-f4e8400c3b2f nodeName:}" failed. No retries permitted until 2026-09-16 14:02:04.606208771 +0000 UTC m=+1158.569567581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-maas-api-tls") pod "maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" (UID: "3748a3db-5c2d-4303-85e3-f4e8400c3b2f") : secret "maas-api-e2e-shared-a-w2-740a25-serving-cert" not found Sep 16 14:02:04.109396 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.109374 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.124859 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.124802 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlvfp\" (UniqueName: \"kubernetes.io/projected/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-kube-api-access-hlvfp\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.610913 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.610874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.613275 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.613252 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:04.792451 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:04.792421 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:05.081107 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:05.081076 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz"] Sep 16 14:02:05.082361 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:02:05.082329 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3748a3db_5c2d_4303_85e3_f4e8400c3b2f.slice/crio-c861642ff4409a5b40f7492a8a23647673966cdfd793c79ec329c07a3a19361e WatchSource:0}: Error finding container c861642ff4409a5b40f7492a8a23647673966cdfd793c79ec329c07a3a19361e: Status 404 returned error can't find the container with id c861642ff4409a5b40f7492a8a23647673966cdfd793c79ec329c07a3a19361e Sep 16 14:02:05.102047 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:05.102019 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" event={"ID":"3748a3db-5c2d-4303-85e3-f4e8400c3b2f","Type":"ContainerStarted","Data":"c861642ff4409a5b40f7492a8a23647673966cdfd793c79ec329c07a3a19361e"} Sep 16 14:02:08.121846 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:08.121806 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" event={"ID":"3748a3db-5c2d-4303-85e3-f4e8400c3b2f","Type":"ContainerStarted","Data":"a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698"} Sep 16 14:02:08.122230 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:08.121984 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:08.145591 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:08.145548 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" podStartSLOduration=2.663243296 podStartE2EDuration="5.145534751s" totalImagesPullingTime="2.482291455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:08.144049594 +0000 UTC m=+1162.107408421" watchObservedRunningTime="2026-09-16 14:02:08.145534751 +0000 UTC m=+1162.108893578" Sep 16 14:02:14.133460 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.133384 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:02:14.573545 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.573480 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6"] Sep 16 14:02:14.573823 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.573793 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" podUID="4dff28b8-676b-44f7-8d39-5a7d25a4bc28" containerName="maas-api" containerID="cri-o://5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2" gracePeriod=30 Sep 16 14:02:14.830796 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.830736 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 14:02:14.912303 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.912266 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-829kp\" (UniqueName: \"kubernetes.io/projected/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-kube-api-access-829kp\") pod \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " Sep 16 14:02:14.912507 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.912327 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-openshift-service-ca\") pod \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " Sep 16 14:02:14.912507 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.912392 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-maas-api-tls\") pod \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " Sep 16 14:02:14.912507 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.912446 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-metrics-certs\") pod \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\" (UID: \"4dff28b8-676b-44f7-8d39-5a7d25a4bc28\") " Sep 16 14:02:14.912777 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.912671 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4dff28b8-676b-44f7-8d39-5a7d25a4bc28" (UID: "4dff28b8-676b-44f7-8d39-5a7d25a4bc28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:02:14.912843 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.912778 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:02:14.914634 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.914602 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4dff28b8-676b-44f7-8d39-5a7d25a4bc28" (UID: "4dff28b8-676b-44f7-8d39-5a7d25a4bc28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:02:14.914739 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.914645 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4dff28b8-676b-44f7-8d39-5a7d25a4bc28" (UID: "4dff28b8-676b-44f7-8d39-5a7d25a4bc28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:02:14.914739 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:14.914676 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-kube-api-access-829kp" (OuterVolumeSpecName: "kube-api-access-829kp") pod "4dff28b8-676b-44f7-8d39-5a7d25a4bc28" (UID: "4dff28b8-676b-44f7-8d39-5a7d25a4bc28"). InnerVolumeSpecName "kube-api-access-829kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:02:15.013727 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.013688 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:02:15.013727 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.013726 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-829kp\" (UniqueName: \"kubernetes.io/projected/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-kube-api-access-829kp\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:02:15.013949 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.013741 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dff28b8-676b-44f7-8d39-5a7d25a4bc28-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:02:15.168090 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.167997 2581 generic.go:356] "Generic (PLEG): container finished" podID="4dff28b8-676b-44f7-8d39-5a7d25a4bc28" containerID="5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2" exitCode=0 Sep 16 14:02:15.168090 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.168046 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" Sep 16 14:02:15.168090 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.168044 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" event={"ID":"4dff28b8-676b-44f7-8d39-5a7d25a4bc28","Type":"ContainerDied","Data":"5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2"} Sep 16 14:02:15.168692 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.168099 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6" event={"ID":"4dff28b8-676b-44f7-8d39-5a7d25a4bc28","Type":"ContainerDied","Data":"5c82c7c6254bd1996267cb80c1f17a4ae88faa4716ed9db363a44637c6172311"} Sep 16 14:02:15.168692 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.168121 2581 scope.go:117] "RemoveContainer" containerID="5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2" Sep 16 14:02:15.180350 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.179495 2581 scope.go:117] "RemoveContainer" containerID="5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2" Sep 16 14:02:15.180835 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:02:15.180805 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2\": container with ID starting with 5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2 not found: ID does not exist" containerID="5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2" Sep 16 14:02:15.180930 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.180846 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2"} err="failed to get container status \"5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2\": rpc error: code = NotFound desc = could not find container \"5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2\": container with ID starting with 5c6f0fbc98b8efad86ee9b84681b8b6900637fde0a31ef6d0fe054b97e3f36c2 not found: ID does not exist" Sep 16 14:02:15.201751 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.201648 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6"] Sep 16 14:02:15.204986 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:15.204963 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b52a89-6bb5d48674-bs9h6"] Sep 16 14:02:16.515740 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:16.515703 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dff28b8-676b-44f7-8d39-5a7d25a4bc28" path="/var/lib/kubelet/pods/4dff28b8-676b-44f7-8d39-5a7d25a4bc28/volumes" Sep 16 14:02:33.460950 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.460913 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk"] Sep 16 14:02:33.461387 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.461199 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" podUID="643f0199-dbb5-42bb-9490-d3118ec8d553" containerName="maas-api" containerID="cri-o://9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf" gracePeriod=30 Sep 16 14:02:33.723981 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.723869 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 14:02:33.896762 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.896737 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-metrics-certs\") pod \"643f0199-dbb5-42bb-9490-d3118ec8d553\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " Sep 16 14:02:33.896967 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.896801 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-maas-api-tls\") pod \"643f0199-dbb5-42bb-9490-d3118ec8d553\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " Sep 16 14:02:33.896967 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.896860 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmjvh\" (UniqueName: \"kubernetes.io/projected/643f0199-dbb5-42bb-9490-d3118ec8d553-kube-api-access-xmjvh\") pod \"643f0199-dbb5-42bb-9490-d3118ec8d553\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " Sep 16 14:02:33.896967 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.896910 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643f0199-dbb5-42bb-9490-d3118ec8d553-openshift-service-ca\") pod \"643f0199-dbb5-42bb-9490-d3118ec8d553\" (UID: \"643f0199-dbb5-42bb-9490-d3118ec8d553\") " Sep 16 14:02:33.897333 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.897298 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643f0199-dbb5-42bb-9490-d3118ec8d553-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "643f0199-dbb5-42bb-9490-d3118ec8d553" (UID: "643f0199-dbb5-42bb-9490-d3118ec8d553"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:02:33.898951 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.898918 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "643f0199-dbb5-42bb-9490-d3118ec8d553" (UID: "643f0199-dbb5-42bb-9490-d3118ec8d553"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:02:33.899126 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.899079 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643f0199-dbb5-42bb-9490-d3118ec8d553-kube-api-access-xmjvh" (OuterVolumeSpecName: "kube-api-access-xmjvh") pod "643f0199-dbb5-42bb-9490-d3118ec8d553" (UID: "643f0199-dbb5-42bb-9490-d3118ec8d553"). InnerVolumeSpecName "kube-api-access-xmjvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:02:33.899303 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.899282 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "643f0199-dbb5-42bb-9490-d3118ec8d553" (UID: "643f0199-dbb5-42bb-9490-d3118ec8d553"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:02:33.964133 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.964101 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn"] Sep 16 14:02:33.964606 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.964587 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dff28b8-676b-44f7-8d39-5a7d25a4bc28" containerName="maas-api" Sep 16 14:02:33.964606 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.964605 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dff28b8-676b-44f7-8d39-5a7d25a4bc28" containerName="maas-api" Sep 16 14:02:33.964762 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.964627 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643f0199-dbb5-42bb-9490-d3118ec8d553" containerName="maas-api" Sep 16 14:02:33.964762 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.964634 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="643f0199-dbb5-42bb-9490-d3118ec8d553" containerName="maas-api" Sep 16 14:02:33.964762 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.964728 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="643f0199-dbb5-42bb-9490-d3118ec8d553" containerName="maas-api" Sep 16 14:02:33.964762 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.964737 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dff28b8-676b-44f7-8d39-5a7d25a4bc28" containerName="maas-api" Sep 16 14:02:33.968087 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.968069 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:33.970465 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.970441 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-9187f5-serving-cert\"" Sep 16 14:02:33.980678 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.980612 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn"] Sep 16 14:02:33.998295 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.998267 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:02:33.998295 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.998293 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643f0199-dbb5-42bb-9490-d3118ec8d553-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:02:33.998444 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.998303 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjvh\" (UniqueName: \"kubernetes.io/projected/643f0199-dbb5-42bb-9490-d3118ec8d553-kube-api-access-xmjvh\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:02:33.998444 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:33.998313 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643f0199-dbb5-42bb-9490-d3118ec8d553-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:02:34.098917 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.098870 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrd2n\" (UniqueName: \"kubernetes.io/projected/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-kube-api-access-xrd2n\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.099068 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.098988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.099113 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.099092 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.099179 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.099112 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.199873 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.199824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.199873 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.199874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.200134 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.199925 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrd2n\" (UniqueName: \"kubernetes.io/projected/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-kube-api-access-xrd2n\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.200134 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.199998 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.200955 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.200921 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.202738 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.202712 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.203161 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.203062 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.211409 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.211385 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrd2n\" (UniqueName: \"kubernetes.io/projected/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-kube-api-access-xrd2n\") pod \"maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.284483 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.284407 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:34.296629 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.296597 2581 generic.go:356] "Generic (PLEG): container finished" podID="643f0199-dbb5-42bb-9490-d3118ec8d553" containerID="9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf" exitCode=0 Sep 16 14:02:34.296740 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.296657 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" Sep 16 14:02:34.296740 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.296690 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" event={"ID":"643f0199-dbb5-42bb-9490-d3118ec8d553","Type":"ContainerDied","Data":"9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf"} Sep 16 14:02:34.296858 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.296773 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk" event={"ID":"643f0199-dbb5-42bb-9490-d3118ec8d553","Type":"ContainerDied","Data":"8084b20b1c297d699f3460d6e79a75a8d34448bdd928ac282a869a0dff4aca1f"} Sep 16 14:02:34.296858 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.296820 2581 scope.go:117] "RemoveContainer" containerID="9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf" Sep 16 14:02:34.309997 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.309617 2581 scope.go:117] "RemoveContainer" containerID="9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf" Sep 16 14:02:34.310565 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:02:34.310132 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf\": container with ID starting with 9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf not found: ID does not exist" containerID="9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf" Sep 16 14:02:34.310565 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.310189 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf"} err="failed to get container status \"9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf\": rpc error: code = NotFound desc = could not find container \"9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf\": container with ID starting with 9b5f3fcac87bddfaad7a4d84c3653de8d24bbfa57391c397cf92f70cc4161fdf not found: ID does not exist" Sep 16 14:02:34.333673 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.332570 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk"] Sep 16 14:02:34.337028 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.337000 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-95743e-585c96497-tkjdk"] Sep 16 14:02:34.517056 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.517020 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643f0199-dbb5-42bb-9490-d3118ec8d553" path="/var/lib/kubelet/pods/643f0199-dbb5-42bb-9490-d3118ec8d553/volumes" Sep 16 14:02:34.573352 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:34.573326 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn"] Sep 16 14:02:34.574523 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:02:34.574501 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd27dd5_c077_44ef_8fb6_7da49b3ae1eb.slice/crio-1f2026df2a311aa0c9cc8db44e0168a32e07f63386822bb32038ed2f4fc90157 WatchSource:0}: Error finding container 1f2026df2a311aa0c9cc8db44e0168a32e07f63386822bb32038ed2f4fc90157: Status 404 returned error can't find the container with id 1f2026df2a311aa0c9cc8db44e0168a32e07f63386822bb32038ed2f4fc90157 Sep 16 14:02:35.304100 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:35.304061 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" event={"ID":"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb","Type":"ContainerStarted","Data":"1f2026df2a311aa0c9cc8db44e0168a32e07f63386822bb32038ed2f4fc90157"} Sep 16 14:02:36.313871 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:36.313790 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" event={"ID":"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb","Type":"ContainerStarted","Data":"af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c"} Sep 16 14:02:36.314301 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:36.313940 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:36.337194 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:36.337128 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" podStartSLOduration=1.92923373 podStartE2EDuration="3.337112074s" totalImagesPullingTime="1.407878344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:36.335223933 +0000 UTC m=+1190.298582770" watchObservedRunningTime="2026-09-16 14:02:36.337112074 +0000 UTC m=+1190.300470903" Sep 16 14:02:42.326461 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:42.326431 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:02:46.591309 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:46.591282 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:02:46.595600 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:46.595573 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:02:46.600149 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:46.600110 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 14:02:46.604776 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:46.604759 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 14:02:51.583391 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.583354 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj"] Sep 16 14:02:51.589098 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.589074 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.591389 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.591362 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-46224d-serving-cert\"" Sep 16 14:02:51.598357 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.598331 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj"] Sep 16 14:02:51.662971 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.662944 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.663111 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.663011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.663111 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.663075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.663242 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.663116 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8qmz\" (UniqueName: \"kubernetes.io/projected/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-kube-api-access-w8qmz\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.764209 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.764171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.764385 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.764253 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.764385 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.764294 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qmz\" (UniqueName: \"kubernetes.io/projected/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-kube-api-access-w8qmz\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.764385 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.764325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.764385 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:02:51.764344 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-serving-cert: secret "maas-api-e2e-shared-a-w3-46224d-serving-cert" not found Sep 16 14:02:51.764611 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:02:51.764428 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls podName:8dfc3bc4-d5f2-470e-86d7-5b41b39ef526 nodeName:}" failed. No retries permitted until 2026-09-16 14:02:52.264404501 +0000 UTC m=+1206.227763323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls") pod "maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" (UID: "8dfc3bc4-d5f2-470e-86d7-5b41b39ef526") : secret "maas-api-e2e-shared-a-w3-46224d-serving-cert" not found Sep 16 14:02:51.764987 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.764965 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.766725 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.766692 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:51.773181 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:51.773163 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qmz\" (UniqueName: \"kubernetes.io/projected/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-kube-api-access-w8qmz\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:52.268328 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:52.268289 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:52.268520 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:02:52.268461 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-serving-cert: secret "maas-api-e2e-shared-a-w3-46224d-serving-cert" not found Sep 16 14:02:52.268596 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:02:52.268553 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls podName:8dfc3bc4-d5f2-470e-86d7-5b41b39ef526 nodeName:}" failed. No retries permitted until 2026-09-16 14:02:53.268533331 +0000 UTC m=+1207.231892141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls") pod "maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" (UID: "8dfc3bc4-d5f2-470e-86d7-5b41b39ef526") : secret "maas-api-e2e-shared-a-w3-46224d-serving-cert" not found Sep 16 14:02:53.277187 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:53.277129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:53.279553 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:53.279527 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:53.402090 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:53.402056 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:53.681709 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:02:53.681048 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfc3bc4_d5f2_470e_86d7_5b41b39ef526.slice/crio-2b9a1bb45eb3d86e251154e866827920058e6d73ed0a356aec75db16bc36d6ea WatchSource:0}: Error finding container 2b9a1bb45eb3d86e251154e866827920058e6d73ed0a356aec75db16bc36d6ea: Status 404 returned error can't find the container with id 2b9a1bb45eb3d86e251154e866827920058e6d73ed0a356aec75db16bc36d6ea Sep 16 14:02:53.681861 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:53.681789 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj"] Sep 16 14:02:54.424500 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:54.424404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" event={"ID":"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526","Type":"ContainerStarted","Data":"c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0"} Sep 16 14:02:54.424500 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:54.424467 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" event={"ID":"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526","Type":"ContainerStarted","Data":"2b9a1bb45eb3d86e251154e866827920058e6d73ed0a356aec75db16bc36d6ea"} Sep 16 14:02:54.424500 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:54.424490 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:02:54.447362 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:54.447315 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" podStartSLOduration=3.058527784 podStartE2EDuration="3.447300655s" totalImagesPullingTime="388.772871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:54.445229958 +0000 UTC m=+1208.408588786" watchObservedRunningTime="2026-09-16 14:02:54.447300655 +0000 UTC m=+1208.410659522" Sep 16 14:02:56.326260 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:56.326228 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-547560"] Sep 16 14:02:56.329941 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:56.329922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-547560" Sep 16 14:02:56.333478 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:56.333461 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f62lv\"" Sep 16 14:02:56.338410 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:56.338385 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-547560"] Sep 16 14:02:56.407792 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:56.407766 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbt7h\" (UniqueName: \"kubernetes.io/projected/9ecfcd4e-ecd3-468d-94f9-2318c89f039b-kube-api-access-hbt7h\") pod \"test-curl-7896-547560\" (UID: \"9ecfcd4e-ecd3-468d-94f9-2318c89f039b\") " pod="opendatahub/test-curl-7896-547560" Sep 16 14:02:56.508941 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:56.508891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbt7h\" (UniqueName: \"kubernetes.io/projected/9ecfcd4e-ecd3-468d-94f9-2318c89f039b-kube-api-access-hbt7h\") pod \"test-curl-7896-547560\" (UID: \"9ecfcd4e-ecd3-468d-94f9-2318c89f039b\") " pod="opendatahub/test-curl-7896-547560" Sep 16 14:02:56.519521 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:56.519497 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbt7h\" (UniqueName: \"kubernetes.io/projected/9ecfcd4e-ecd3-468d-94f9-2318c89f039b-kube-api-access-hbt7h\") pod \"test-curl-7896-547560\" (UID: \"9ecfcd4e-ecd3-468d-94f9-2318c89f039b\") " pod="opendatahub/test-curl-7896-547560" Sep 16 14:02:56.640458 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:56.640383 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-547560" Sep 16 14:02:56.918423 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:56.918398 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-547560"] Sep 16 14:02:56.919638 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:02:56.919606 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ecfcd4e_ecd3_468d_94f9_2318c89f039b.slice/crio-e144d489547b4174d8ef06fdebad20e43e985a27d09d4cc3d4b91b1844e5e09b WatchSource:0}: Error finding container e144d489547b4174d8ef06fdebad20e43e985a27d09d4cc3d4b91b1844e5e09b: Status 404 returned error can't find the container with id e144d489547b4174d8ef06fdebad20e43e985a27d09d4cc3d4b91b1844e5e09b Sep 16 14:02:57.443328 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:02:57.443292 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-547560" event={"ID":"9ecfcd4e-ecd3-468d-94f9-2318c89f039b","Type":"ContainerStarted","Data":"e144d489547b4174d8ef06fdebad20e43e985a27d09d4cc3d4b91b1844e5e09b"} Sep 16 14:03:00.436016 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:00.435938 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:03:00.465608 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:00.465571 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-547560" event={"ID":"9ecfcd4e-ecd3-468d-94f9-2318c89f039b","Type":"ContainerStarted","Data":"99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925"} Sep 16 14:03:00.483266 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:00.483214 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7896-547560" podStartSLOduration=1.372252611 podStartE2EDuration="4.483199402s" totalImagesPullingTime="3.110946791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:00.480846392 +0000 UTC m=+1214.444205218" watchObservedRunningTime="2026-09-16 14:03:00.483199402 +0000 UTC m=+1214.446558229" Sep 16 14:03:01.198617 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:01.198534 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-547560"] Sep 16 14:03:01.205453 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:01.205423 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-547560"] Sep 16 14:03:01.472293 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:01.472262 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7896-547560" container="test-curl-7896-547560" Sep 16 14:03:02.479405 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:02.479343 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7896-547560" podUID="9ecfcd4e-ecd3-468d-94f9-2318c89f039b" containerName="test-curl-7896-547560" containerID="cri-o://99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925" gracePeriod=2 Sep 16 14:03:02.482987 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:02.482955 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7896-547560\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="9ecfcd4e-ecd3-468d-94f9-2318c89f039b" pod="opendatahub/test-curl-7896-547560" Sep 16 14:03:03.321300 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.321269 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-2c1b5d"] Sep 16 14:03:03.321713 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.321700 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ecfcd4e-ecd3-468d-94f9-2318c89f039b" containerName="test-curl-7896-547560" Sep 16 14:03:03.321770 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.321714 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ecfcd4e-ecd3-468d-94f9-2318c89f039b" containerName="test-curl-7896-547560" Sep 16 14:03:03.321809 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.321800 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ecfcd4e-ecd3-468d-94f9-2318c89f039b" containerName="test-curl-7896-547560" Sep 16 14:03:03.325152 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.325125 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-2c1b5d" Sep 16 14:03:03.332533 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.332508 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-2c1b5d"] Sep 16 14:03:03.377266 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.377234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g942\" (UniqueName: \"kubernetes.io/projected/8fa9ed10-0eb6-4804-b78c-5e39b465cf30-kube-api-access-2g942\") pod \"test-curl-7896-2c1b5d\" (UID: \"8fa9ed10-0eb6-4804-b78c-5e39b465cf30\") " pod="opendatahub/test-curl-7896-2c1b5d" Sep 16 14:03:03.478818 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.478780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g942\" (UniqueName: \"kubernetes.io/projected/8fa9ed10-0eb6-4804-b78c-5e39b465cf30-kube-api-access-2g942\") pod \"test-curl-7896-2c1b5d\" (UID: \"8fa9ed10-0eb6-4804-b78c-5e39b465cf30\") " pod="opendatahub/test-curl-7896-2c1b5d" Sep 16 14:03:03.488945 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.488918 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g942\" (UniqueName: \"kubernetes.io/projected/8fa9ed10-0eb6-4804-b78c-5e39b465cf30-kube-api-access-2g942\") pod \"test-curl-7896-2c1b5d\" (UID: \"8fa9ed10-0eb6-4804-b78c-5e39b465cf30\") " pod="opendatahub/test-curl-7896-2c1b5d" Sep 16 14:03:03.635406 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.635309 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-2c1b5d" Sep 16 14:03:03.925359 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:03.925320 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-2c1b5d"] Sep 16 14:03:03.926960 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:03:03.926917 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fa9ed10_0eb6_4804_b78c_5e39b465cf30.slice/crio-6c1072b207a84f416e1adaee40d40d3596d5434e65bbd7c2f2c79c80e0be1e6f WatchSource:0}: Error finding container 6c1072b207a84f416e1adaee40d40d3596d5434e65bbd7c2f2c79c80e0be1e6f: Status 404 returned error can't find the container with id 6c1072b207a84f416e1adaee40d40d3596d5434e65bbd7c2f2c79c80e0be1e6f Sep 16 14:03:04.493598 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:04.493559 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-2c1b5d" event={"ID":"8fa9ed10-0eb6-4804-b78c-5e39b465cf30","Type":"ContainerStarted","Data":"a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2"} Sep 16 14:03:04.493968 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:04.493611 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-2c1b5d" event={"ID":"8fa9ed10-0eb6-4804-b78c-5e39b465cf30","Type":"ContainerStarted","Data":"6c1072b207a84f416e1adaee40d40d3596d5434e65bbd7c2f2c79c80e0be1e6f"} Sep 16 14:03:04.514586 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:04.514541 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7896-2c1b5d" podStartSLOduration=1.514526008 podStartE2EDuration="1.514526008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:04.511448037 +0000 UTC m=+1218.474806865" watchObservedRunningTime="2026-09-16 14:03:04.514526008 +0000 UTC m=+1218.477884835" Sep 16 14:03:04.670448 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:04.670425 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-547560" Sep 16 14:03:04.791242 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:04.791167 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbt7h\" (UniqueName: \"kubernetes.io/projected/9ecfcd4e-ecd3-468d-94f9-2318c89f039b-kube-api-access-hbt7h\") pod \"9ecfcd4e-ecd3-468d-94f9-2318c89f039b\" (UID: \"9ecfcd4e-ecd3-468d-94f9-2318c89f039b\") " Sep 16 14:03:04.793060 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:04.793036 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecfcd4e-ecd3-468d-94f9-2318c89f039b-kube-api-access-hbt7h" (OuterVolumeSpecName: "kube-api-access-hbt7h") pod "9ecfcd4e-ecd3-468d-94f9-2318c89f039b" (UID: "9ecfcd4e-ecd3-468d-94f9-2318c89f039b"). InnerVolumeSpecName "kube-api-access-hbt7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:03:04.893133 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:04.893102 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbt7h\" (UniqueName: \"kubernetes.io/projected/9ecfcd4e-ecd3-468d-94f9-2318c89f039b-kube-api-access-hbt7h\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:03:05.193038 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:05.192961 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-2c1b5d"] Sep 16 14:03:05.197714 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:05.197687 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-2c1b5d"] Sep 16 14:03:05.500948 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:05.500917 2581 generic.go:356] "Generic (PLEG): container finished" podID="9ecfcd4e-ecd3-468d-94f9-2318c89f039b" containerID="99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925" exitCode=137 Sep 16 14:03:05.501359 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:05.500958 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-547560" Sep 16 14:03:05.501359 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:05.501019 2581 scope.go:117] "RemoveContainer" containerID="99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925" Sep 16 14:03:05.501479 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:05.501360 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7896-2c1b5d" container="test-curl-7896-2c1b5d" Sep 16 14:03:05.510723 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:05.510707 2581 scope.go:117] "RemoveContainer" containerID="99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925" Sep 16 14:03:05.511009 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:05.510989 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925\": container with ID starting with 99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925 not found: ID does not exist" containerID="99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925" Sep 16 14:03:05.511063 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:05.511016 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925"} err="failed to get container status \"99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925\": rpc error: code = NotFound desc = could not find container \"99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925\": container with ID starting with 99a4d8fecbbea18ae036181e2f8a4aef2e4c85cdf42c242c79f0eaecaa525925 not found: ID does not exist" Sep 16 14:03:05.511700 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:05.511676 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7896-547560\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="9ecfcd4e-ecd3-468d-94f9-2318c89f039b" pod="opendatahub/test-curl-7896-547560" Sep 16 14:03:06.512090 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:06.512050 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7896-2c1b5d" podUID="8fa9ed10-0eb6-4804-b78c-5e39b465cf30" containerName="test-curl-7896-2c1b5d" containerID="cri-o://a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2" gracePeriod=2 Sep 16 14:03:06.514478 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:06.514451 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7896-547560\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="9ecfcd4e-ecd3-468d-94f9-2318c89f039b" pod="opendatahub/test-curl-7896-547560" Sep 16 14:03:06.516035 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:06.516012 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ecfcd4e-ecd3-468d-94f9-2318c89f039b" path="/var/lib/kubelet/pods/9ecfcd4e-ecd3-468d-94f9-2318c89f039b/volumes" Sep 16 14:03:06.518323 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:06.518302 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7896-2c1b5d\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="8fa9ed10-0eb6-4804-b78c-5e39b465cf30" pod="opendatahub/test-curl-7896-2c1b5d" Sep 16 14:03:06.520198 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:06.520171 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7896-547560\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="9ecfcd4e-ecd3-468d-94f9-2318c89f039b" pod="opendatahub/test-curl-7896-547560" Sep 16 14:03:08.662109 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:08.662075 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-2c1b5d" Sep 16 14:03:08.733258 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:08.733218 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g942\" (UniqueName: \"kubernetes.io/projected/8fa9ed10-0eb6-4804-b78c-5e39b465cf30-kube-api-access-2g942\") pod \"8fa9ed10-0eb6-4804-b78c-5e39b465cf30\" (UID: \"8fa9ed10-0eb6-4804-b78c-5e39b465cf30\") " Sep 16 14:03:08.735220 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:08.735193 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa9ed10-0eb6-4804-b78c-5e39b465cf30-kube-api-access-2g942" (OuterVolumeSpecName: "kube-api-access-2g942") pod "8fa9ed10-0eb6-4804-b78c-5e39b465cf30" (UID: "8fa9ed10-0eb6-4804-b78c-5e39b465cf30"). InnerVolumeSpecName "kube-api-access-2g942". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:03:08.834427 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:08.834347 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g942\" (UniqueName: \"kubernetes.io/projected/8fa9ed10-0eb6-4804-b78c-5e39b465cf30-kube-api-access-2g942\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:03:09.529569 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:09.529534 2581 generic.go:356] "Generic (PLEG): container finished" podID="8fa9ed10-0eb6-4804-b78c-5e39b465cf30" containerID="a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2" exitCode=137 Sep 16 14:03:09.529736 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:09.529576 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-2c1b5d" Sep 16 14:03:09.529736 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:09.529592 2581 scope.go:117] "RemoveContainer" containerID="a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2" Sep 16 14:03:09.539548 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:09.539529 2581 scope.go:117] "RemoveContainer" containerID="a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2" Sep 16 14:03:09.539822 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:09.539805 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2\": container with ID starting with a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2 not found: ID does not exist" containerID="a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2" Sep 16 14:03:09.539868 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:09.539832 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2"} err="failed to get container status \"a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2\": rpc error: code = NotFound desc = could not find container \"a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2\": container with ID starting with a43acee6c12314c48ae4416d7a752b36983988ce518d5afb96544778df6df2e2 not found: ID does not exist" Sep 16 14:03:09.540127 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:09.540105 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7896-2c1b5d\" is forbidden: User \"system:node:ip-10-0-143-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-222.ec2.internal' and this object" podUID="8fa9ed10-0eb6-4804-b78c-5e39b465cf30" pod="opendatahub/test-curl-7896-2c1b5d" Sep 16 14:03:10.522405 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:10.522369 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa9ed10-0eb6-4804-b78c-5e39b465cf30" path="/var/lib/kubelet/pods/8fa9ed10-0eb6-4804-b78c-5e39b465cf30/volumes" Sep 16 14:03:15.089659 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.089625 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw"] Sep 16 14:03:15.090276 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.090256 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa9ed10-0eb6-4804-b78c-5e39b465cf30" containerName="test-curl-7896-2c1b5d" Sep 16 14:03:15.090276 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.090277 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa9ed10-0eb6-4804-b78c-5e39b465cf30" containerName="test-curl-7896-2c1b5d" Sep 16 14:03:15.090427 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.090411 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa9ed10-0eb6-4804-b78c-5e39b465cf30" containerName="test-curl-7896-2c1b5d" Sep 16 14:03:15.095030 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.095008 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.097769 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.097736 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ae7c3682-serving-cert\"" Sep 16 14:03:15.108980 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.108958 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw"] Sep 16 14:03:15.199401 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.199365 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9978fa-99fc-4390-9ea5-bb63db314f26-openshift-service-ca\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.199602 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.199422 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-maas-api-tls\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.199602 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.199526 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvs7\" (UniqueName: \"kubernetes.io/projected/cf9978fa-99fc-4390-9ea5-bb63db314f26-kube-api-access-8fvs7\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.199723 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.199630 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-metrics-certs\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.301063 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.301030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9978fa-99fc-4390-9ea5-bb63db314f26-openshift-service-ca\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.301063 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.301065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-maas-api-tls\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.301299 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.301094 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvs7\" (UniqueName: \"kubernetes.io/projected/cf9978fa-99fc-4390-9ea5-bb63db314f26-kube-api-access-8fvs7\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.301299 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.301181 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-metrics-certs\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.301795 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.301774 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9978fa-99fc-4390-9ea5-bb63db314f26-openshift-service-ca\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.303677 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.303655 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-metrics-certs\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.303761 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.303692 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-maas-api-tls\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.319195 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.319167 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvs7\" (UniqueName: \"kubernetes.io/projected/cf9978fa-99fc-4390-9ea5-bb63db314f26-kube-api-access-8fvs7\") pod \"maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.405531 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.405470 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:15.704760 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:15.704733 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw"] Sep 16 14:03:15.705614 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:03:15.705592 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9978fa_99fc_4390_9ea5_bb63db314f26.slice/crio-adf7b9168a03e6f5dabcbefe8e099ef31d1b409ca3a0947e7301903a90b4182f WatchSource:0}: Error finding container adf7b9168a03e6f5dabcbefe8e099ef31d1b409ca3a0947e7301903a90b4182f: Status 404 returned error can't find the container with id adf7b9168a03e6f5dabcbefe8e099ef31d1b409ca3a0947e7301903a90b4182f Sep 16 14:03:16.576989 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:16.576948 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" event={"ID":"cf9978fa-99fc-4390-9ea5-bb63db314f26","Type":"ContainerStarted","Data":"adf7b9168a03e6f5dabcbefe8e099ef31d1b409ca3a0947e7301903a90b4182f"} Sep 16 14:03:17.585516 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:17.585485 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" event={"ID":"cf9978fa-99fc-4390-9ea5-bb63db314f26","Type":"ContainerStarted","Data":"d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11"} Sep 16 14:03:17.585914 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:17.585615 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:17.625209 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:17.625154 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" podStartSLOduration=1.240409204 podStartE2EDuration="2.625121315s" totalImagesPullingTime="1.384712111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:17.620020425 +0000 UTC m=+1231.583379254" watchObservedRunningTime="2026-09-16 14:03:17.625121315 +0000 UTC m=+1231.588480143" Sep 16 14:03:23.597517 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:23.597490 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:03:23.899034 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:23.898947 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs"] Sep 16 14:03:23.902650 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:23.902628 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:23.906210 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:23.906184 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-b4884e-serving-cert\"" Sep 16 14:03:23.918936 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:23.918912 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs"] Sep 16 14:03:23.983325 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:23.983297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:23.983489 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:23.983340 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:23.983489 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:23.983447 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:23.983621 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:23.983498 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5dkx\" (UniqueName: \"kubernetes.io/projected/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-kube-api-access-w5dkx\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:24.084204 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:24.084171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:24.084380 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:24.084233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:24.084380 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:24.084282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:24.084504 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:24.084445 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-serving-cert: secret "maas-api-e2e-shared-b-w3-b4884e-serving-cert" not found Sep 16 14:03:24.084504 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:24.084471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5dkx\" (UniqueName: \"kubernetes.io/projected/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-kube-api-access-w5dkx\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:24.084622 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:03:24.084523 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-maas-api-tls podName:9df2fc3d-cc16-42e4-9d81-5b436fe22fa4 nodeName:}" failed. No retries permitted until 2026-09-16 14:03:24.584501881 +0000 UTC m=+1238.547860703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-maas-api-tls") pod "maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" (UID: "9df2fc3d-cc16-42e4-9d81-5b436fe22fa4") : secret "maas-api-e2e-shared-b-w3-b4884e-serving-cert" not found Sep 16 14:03:24.085070 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:24.085045 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:24.086738 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:24.086718 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:24.101514 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:24.101495 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5dkx\" (UniqueName: \"kubernetes.io/projected/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-kube-api-access-w5dkx\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:24.588192 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:24.588129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:24.590551 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:24.590528 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:24.813012 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:24.812981 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:25.099603 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:25.099573 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs"] Sep 16 14:03:25.100781 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:03:25.100750 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df2fc3d_cc16_42e4_9d81_5b436fe22fa4.slice/crio-9700aed1a8942582eae8c4924220c41ecf4eff6f47cea5047ecd7cd55ef55fbd WatchSource:0}: Error finding container 9700aed1a8942582eae8c4924220c41ecf4eff6f47cea5047ecd7cd55ef55fbd: Status 404 returned error can't find the container with id 9700aed1a8942582eae8c4924220c41ecf4eff6f47cea5047ecd7cd55ef55fbd Sep 16 14:03:25.634379 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:25.634337 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" event={"ID":"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4","Type":"ContainerStarted","Data":"405d2933155a74f53f9f7a1ece6ce35ef821a404d1dec0103c07f8e3261cc4b3"} Sep 16 14:03:25.634522 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:25.634407 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:03:25.634522 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:25.634433 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" event={"ID":"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4","Type":"ContainerStarted","Data":"9700aed1a8942582eae8c4924220c41ecf4eff6f47cea5047ecd7cd55ef55fbd"} Sep 16 14:03:25.659620 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:25.659575 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" podStartSLOduration=2.269005895 podStartE2EDuration="2.659559624s" totalImagesPullingTime="390.553729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:25.657241731 +0000 UTC m=+1239.620600586" watchObservedRunningTime="2026-09-16 14:03:25.659559624 +0000 UTC m=+1239.622918486" Sep 16 14:03:31.645622 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:03:31.645593 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:04:47.092721 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:04:47.092689 2581 scope.go:117] "RemoveContainer" containerID="7c8d62d196681664f62f6ded950ba3e94b9a56971761345e52f98696f9c61c6f" Sep 16 14:05:01.302273 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.302238 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw"] Sep 16 14:05:01.302667 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.302447 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" podUID="cf9978fa-99fc-4390-9ea5-bb63db314f26" containerName="maas-api" containerID="cri-o://d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11" gracePeriod=30 Sep 16 14:05:01.582482 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.582448 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:05:01.613833 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.613795 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9978fa-99fc-4390-9ea5-bb63db314f26-openshift-service-ca\") pod \"cf9978fa-99fc-4390-9ea5-bb63db314f26\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " Sep 16 14:05:01.614018 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.613873 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-metrics-certs\") pod \"cf9978fa-99fc-4390-9ea5-bb63db314f26\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " Sep 16 14:05:01.614018 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.613901 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-maas-api-tls\") pod \"cf9978fa-99fc-4390-9ea5-bb63db314f26\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " Sep 16 14:05:01.614018 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.613922 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fvs7\" (UniqueName: \"kubernetes.io/projected/cf9978fa-99fc-4390-9ea5-bb63db314f26-kube-api-access-8fvs7\") pod \"cf9978fa-99fc-4390-9ea5-bb63db314f26\" (UID: \"cf9978fa-99fc-4390-9ea5-bb63db314f26\") " Sep 16 14:05:01.614326 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.614300 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9978fa-99fc-4390-9ea5-bb63db314f26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf9978fa-99fc-4390-9ea5-bb63db314f26" (UID: "cf9978fa-99fc-4390-9ea5-bb63db314f26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:05:01.616106 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.616072 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf9978fa-99fc-4390-9ea5-bb63db314f26" (UID: "cf9978fa-99fc-4390-9ea5-bb63db314f26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:05:01.616283 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.616106 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cf9978fa-99fc-4390-9ea5-bb63db314f26" (UID: "cf9978fa-99fc-4390-9ea5-bb63db314f26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:05:01.616283 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.616251 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9978fa-99fc-4390-9ea5-bb63db314f26-kube-api-access-8fvs7" (OuterVolumeSpecName: "kube-api-access-8fvs7") pod "cf9978fa-99fc-4390-9ea5-bb63db314f26" (UID: "cf9978fa-99fc-4390-9ea5-bb63db314f26"). InnerVolumeSpecName "kube-api-access-8fvs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:05:01.715825 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.715774 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9978fa-99fc-4390-9ea5-bb63db314f26-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:05:01.715825 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.715829 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:05:01.716035 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.715844 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf9978fa-99fc-4390-9ea5-bb63db314f26-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:05:01.716035 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:01.715858 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fvs7\" (UniqueName: \"kubernetes.io/projected/cf9978fa-99fc-4390-9ea5-bb63db314f26-kube-api-access-8fvs7\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:05:02.234275 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:02.234244 2581 generic.go:356] "Generic (PLEG): container finished" podID="cf9978fa-99fc-4390-9ea5-bb63db314f26" containerID="d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11" exitCode=0 Sep 16 14:05:02.234455 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:02.234292 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" Sep 16 14:05:02.234455 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:02.234330 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" event={"ID":"cf9978fa-99fc-4390-9ea5-bb63db314f26","Type":"ContainerDied","Data":"d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11"} Sep 16 14:05:02.234455 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:02.234408 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw" event={"ID":"cf9978fa-99fc-4390-9ea5-bb63db314f26","Type":"ContainerDied","Data":"adf7b9168a03e6f5dabcbefe8e099ef31d1b409ca3a0947e7301903a90b4182f"} Sep 16 14:05:02.234455 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:02.234431 2581 scope.go:117] "RemoveContainer" containerID="d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11" Sep 16 14:05:02.244955 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:02.244937 2581 scope.go:117] "RemoveContainer" containerID="d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11" Sep 16 14:05:02.245397 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:05:02.245331 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11\": container with ID starting with d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11 not found: ID does not exist" containerID="d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11" Sep 16 14:05:02.245397 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:02.245377 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11"} err="failed to get container status \"d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11\": rpc error: code = NotFound desc = could not find container \"d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11\": container with ID starting with d5f17a208edda21c59190a2db7d03880a83f6f6dcb37797785051ed74199de11 not found: ID does not exist" Sep 16 14:05:02.270236 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:02.270176 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw"] Sep 16 14:05:02.276761 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:02.276726 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae7c3682-7bf5d96d5b-jxflw"] Sep 16 14:05:02.517029 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:02.516940 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf9978fa-99fc-4390-9ea5-bb63db314f26" path="/var/lib/kubelet/pods/cf9978fa-99fc-4390-9ea5-bb63db314f26/volumes" Sep 16 14:05:47.118303 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:05:47.118268 2581 scope.go:117] "RemoveContainer" containerID="4927a098cc1a0d815407515c113f16f0b3e060aa22d9938d741fe1a7d9b0f820" Sep 16 14:06:39.319673 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.319599 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg"] Sep 16 14:06:39.320073 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.319790 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" podUID="71540276-4d70-40ce-a0b6-7d54fa468543" containerName="maas-api" containerID="cri-o://711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f" gracePeriod=30 Sep 16 14:06:39.581825 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.581773 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 14:06:39.731627 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.731590 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-metrics-certs\") pod \"71540276-4d70-40ce-a0b6-7d54fa468543\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " Sep 16 14:06:39.731812 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.731650 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71540276-4d70-40ce-a0b6-7d54fa468543-openshift-service-ca\") pod \"71540276-4d70-40ce-a0b6-7d54fa468543\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " Sep 16 14:06:39.731812 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.731751 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvfmv\" (UniqueName: \"kubernetes.io/projected/71540276-4d70-40ce-a0b6-7d54fa468543-kube-api-access-bvfmv\") pod \"71540276-4d70-40ce-a0b6-7d54fa468543\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " Sep 16 14:06:39.731812 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.731778 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-maas-api-tls\") pod \"71540276-4d70-40ce-a0b6-7d54fa468543\" (UID: \"71540276-4d70-40ce-a0b6-7d54fa468543\") " Sep 16 14:06:39.732036 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.732005 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71540276-4d70-40ce-a0b6-7d54fa468543-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71540276-4d70-40ce-a0b6-7d54fa468543" (UID: "71540276-4d70-40ce-a0b6-7d54fa468543"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:06:39.732165 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.732120 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71540276-4d70-40ce-a0b6-7d54fa468543-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:39.734623 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.734564 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71540276-4d70-40ce-a0b6-7d54fa468543-kube-api-access-bvfmv" (OuterVolumeSpecName: "kube-api-access-bvfmv") pod "71540276-4d70-40ce-a0b6-7d54fa468543" (UID: "71540276-4d70-40ce-a0b6-7d54fa468543"). InnerVolumeSpecName "kube-api-access-bvfmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:39.734783 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.734667 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71540276-4d70-40ce-a0b6-7d54fa468543" (UID: "71540276-4d70-40ce-a0b6-7d54fa468543"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:06:39.734783 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.734688 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71540276-4d70-40ce-a0b6-7d54fa468543" (UID: "71540276-4d70-40ce-a0b6-7d54fa468543"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:06:39.832717 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.832632 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:39.832717 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.832666 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvfmv\" (UniqueName: \"kubernetes.io/projected/71540276-4d70-40ce-a0b6-7d54fa468543-kube-api-access-bvfmv\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:39.832717 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.832679 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71540276-4d70-40ce-a0b6-7d54fa468543-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:39.834064 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.834035 2581 generic.go:356] "Generic (PLEG): container finished" podID="71540276-4d70-40ce-a0b6-7d54fa468543" containerID="711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f" exitCode=0 Sep 16 14:06:39.834192 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.834082 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" Sep 16 14:06:39.834192 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.834120 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" event={"ID":"71540276-4d70-40ce-a0b6-7d54fa468543","Type":"ContainerDied","Data":"711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f"} Sep 16 14:06:39.834192 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.834189 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg" event={"ID":"71540276-4d70-40ce-a0b6-7d54fa468543","Type":"ContainerDied","Data":"f748f09cfac78b856ae78f6ed6259c6658061a40e0cf09666311a9e515e1b300"} Sep 16 14:06:39.834309 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.834213 2581 scope.go:117] "RemoveContainer" containerID="711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f" Sep 16 14:06:39.844530 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.844515 2581 scope.go:117] "RemoveContainer" containerID="711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f" Sep 16 14:06:39.844783 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:06:39.844767 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f\": container with ID starting with 711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f not found: ID does not exist" containerID="711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f" Sep 16 14:06:39.844848 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.844789 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f"} err="failed to get container status \"711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f\": rpc error: code = NotFound desc = could not find container \"711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f\": container with ID starting with 711ec327054458d8ff5ca667a680ba735e58653280defb66496cac0ed81c8e7f not found: ID does not exist" Sep 16 14:06:39.870721 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.870697 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg"] Sep 16 14:06:39.873909 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:39.873886 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035c64-dbc69b869-j6mcg"] Sep 16 14:06:40.516438 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:40.516408 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71540276-4d70-40ce-a0b6-7d54fa468543" path="/var/lib/kubelet/pods/71540276-4d70-40ce-a0b6-7d54fa468543/volumes" Sep 16 14:06:47.670809 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:47.670777 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj"] Sep 16 14:06:47.671184 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:47.671015 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" containerID="cri-o://50db27b94738fea29b0425aa5dac6937b77c654fbc724db9cc1a81461380abc4" gracePeriod=30 Sep 16 14:06:52.914638 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:52.914608 2581 generic.go:356] "Generic (PLEG): container finished" podID="34473024-9c65-41aa-9007-5ddbc65490eb" containerID="50db27b94738fea29b0425aa5dac6937b77c654fbc724db9cc1a81461380abc4" exitCode=0 Sep 16 14:06:52.914989 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:52.914679 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" event={"ID":"34473024-9c65-41aa-9007-5ddbc65490eb","Type":"ContainerDied","Data":"50db27b94738fea29b0425aa5dac6937b77c654fbc724db9cc1a81461380abc4"} Sep 16 14:06:52.914989 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:52.914739 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" event={"ID":"34473024-9c65-41aa-9007-5ddbc65490eb","Type":"ContainerDied","Data":"186f874aa112cb2f51ba1708d47da97b0f0b1a8526d5584022282de88bda1b34"} Sep 16 14:06:52.914989 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:52.914760 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="186f874aa112cb2f51ba1708d47da97b0f0b1a8526d5584022282de88bda1b34" Sep 16 14:06:52.914989 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:52.914791 2581 scope.go:117] "RemoveContainer" containerID="f70d5ca339cd189c088c91a92dca490e4bd5b91f8c4ca32206e97f66eb4c6aea" Sep 16 14:06:52.926471 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:52.926450 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 14:06:53.065922 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.065891 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-data\") pod \"34473024-9c65-41aa-9007-5ddbc65490eb\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " Sep 16 14:06:53.066097 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.065955 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-certs\") pod \"34473024-9c65-41aa-9007-5ddbc65490eb\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " Sep 16 14:06:53.066097 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.065993 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-credential-socket\") pod \"34473024-9c65-41aa-9007-5ddbc65490eb\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " Sep 16 14:06:53.066097 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066013 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34473024-9c65-41aa-9007-5ddbc65490eb-istio-podinfo\") pod \"34473024-9c65-41aa-9007-5ddbc65490eb\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " Sep 16 14:06:53.066097 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066043 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-envoy\") pod \"34473024-9c65-41aa-9007-5ddbc65490eb\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " Sep 16 14:06:53.066097 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066076 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34473024-9c65-41aa-9007-5ddbc65490eb-istiod-ca-cert\") pod \"34473024-9c65-41aa-9007-5ddbc65490eb\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " Sep 16 14:06:53.066404 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066127 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-socket\") pod \"34473024-9c65-41aa-9007-5ddbc65490eb\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " Sep 16 14:06:53.066404 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066191 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-istio-token\") pod \"34473024-9c65-41aa-9007-5ddbc65490eb\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " Sep 16 14:06:53.066404 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066216 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "34473024-9c65-41aa-9007-5ddbc65490eb" (UID: "34473024-9c65-41aa-9007-5ddbc65490eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:53.066404 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066279 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n69wk\" (UniqueName: \"kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-kube-api-access-n69wk\") pod \"34473024-9c65-41aa-9007-5ddbc65490eb\" (UID: \"34473024-9c65-41aa-9007-5ddbc65490eb\") " Sep 16 14:06:53.066608 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066405 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34473024-9c65-41aa-9007-5ddbc65490eb" (UID: "34473024-9c65-41aa-9007-5ddbc65490eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:53.066608 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066532 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34473024-9c65-41aa-9007-5ddbc65490eb" (UID: "34473024-9c65-41aa-9007-5ddbc65490eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:53.066693 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066622 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34473024-9c65-41aa-9007-5ddbc65490eb" (UID: "34473024-9c65-41aa-9007-5ddbc65490eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:53.066693 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066651 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34473024-9c65-41aa-9007-5ddbc65490eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34473024-9c65-41aa-9007-5ddbc65490eb" (UID: "34473024-9c65-41aa-9007-5ddbc65490eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:06:53.066809 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066733 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-socket\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:53.066809 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066750 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-data\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:53.066809 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066759 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-workload-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:53.066809 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066767 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-credential-socket\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:53.066809 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.066775 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34473024-9c65-41aa-9007-5ddbc65490eb-istiod-ca-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:53.068499 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.068476 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "34473024-9c65-41aa-9007-5ddbc65490eb" (UID: "34473024-9c65-41aa-9007-5ddbc65490eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:53.068637 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.068523 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34473024-9c65-41aa-9007-5ddbc65490eb" (UID: "34473024-9c65-41aa-9007-5ddbc65490eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 14:06:53.068808 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.068786 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34473024-9c65-41aa-9007-5ddbc65490eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34473024-9c65-41aa-9007-5ddbc65490eb" (UID: "34473024-9c65-41aa-9007-5ddbc65490eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 14:06:53.068924 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.068891 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-kube-api-access-n69wk" (OuterVolumeSpecName: "kube-api-access-n69wk") pod "34473024-9c65-41aa-9007-5ddbc65490eb" (UID: "34473024-9c65-41aa-9007-5ddbc65490eb"). InnerVolumeSpecName "kube-api-access-n69wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:06:53.167970 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.167940 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-istio-token\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:53.167970 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.167966 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n69wk\" (UniqueName: \"kubernetes.io/projected/34473024-9c65-41aa-9007-5ddbc65490eb-kube-api-access-n69wk\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:53.167970 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.167976 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34473024-9c65-41aa-9007-5ddbc65490eb-istio-podinfo\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:53.168208 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.167986 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34473024-9c65-41aa-9007-5ddbc65490eb-istio-envoy\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:06:53.922403 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.922370 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj" Sep 16 14:06:53.984878 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.984845 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj"] Sep 16 14:06:53.989337 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:53.989309 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035c64-openshift-default-5c74f5c87d-679jj"] Sep 16 14:06:54.516674 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:06:54.516643 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" path="/var/lib/kubelet/pods/34473024-9c65-41aa-9007-5ddbc65490eb/volumes" Sep 16 14:07:12.140996 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.140964 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz"] Sep 16 14:07:12.141468 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141433 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71540276-4d70-40ce-a0b6-7d54fa468543" containerName="maas-api" Sep 16 14:07:12.141468 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141445 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="71540276-4d70-40ce-a0b6-7d54fa468543" containerName="maas-api" Sep 16 14:07:12.141468 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141463 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" Sep 16 14:07:12.141468 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141468 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" Sep 16 14:07:12.141617 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141478 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" Sep 16 14:07:12.141617 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141485 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" Sep 16 14:07:12.141617 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141492 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9978fa-99fc-4390-9ea5-bb63db314f26" containerName="maas-api" Sep 16 14:07:12.141617 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141497 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9978fa-99fc-4390-9ea5-bb63db314f26" containerName="maas-api" Sep 16 14:07:12.141617 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141564 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" Sep 16 14:07:12.141617 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141572 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34473024-9c65-41aa-9007-5ddbc65490eb" containerName="istio-proxy" Sep 16 14:07:12.141617 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141578 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9978fa-99fc-4390-9ea5-bb63db314f26" containerName="maas-api" Sep 16 14:07:12.141617 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.141585 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="71540276-4d70-40ce-a0b6-7d54fa468543" containerName="maas-api" Sep 16 14:07:12.146051 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.146033 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.149329 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.149310 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7f75a3-serving-cert\"" Sep 16 14:07:12.168950 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.168927 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz"] Sep 16 14:07:12.250178 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.250128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7lj\" (UniqueName: \"kubernetes.io/projected/161d9559-3bfa-4713-92f5-1e5764240302-kube-api-access-hn7lj\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.250178 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.250184 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-metrics-certs\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.250381 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.250267 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/161d9559-3bfa-4713-92f5-1e5764240302-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.250381 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.250329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-maas-api-tls\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.350908 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.350877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7lj\" (UniqueName: \"kubernetes.io/projected/161d9559-3bfa-4713-92f5-1e5764240302-kube-api-access-hn7lj\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.350908 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.350915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-metrics-certs\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.351136 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.351062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/161d9559-3bfa-4713-92f5-1e5764240302-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.351241 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.351186 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-maas-api-tls\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.351708 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.351683 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/161d9559-3bfa-4713-92f5-1e5764240302-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.353420 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.353397 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-maas-api-tls\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.353511 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.353450 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-metrics-certs\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.364600 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.364578 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7lj\" (UniqueName: \"kubernetes.io/projected/161d9559-3bfa-4713-92f5-1e5764240302-kube-api-access-hn7lj\") pod \"maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.456983 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.456895 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:12.754331 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:07:12.754293 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161d9559_3bfa_4713_92f5_1e5764240302.slice/crio-9664ce6804b382ec825630da5fbd42112167c597568f1b81e5390efa41b8540c WatchSource:0}: Error finding container 9664ce6804b382ec825630da5fbd42112167c597568f1b81e5390efa41b8540c: Status 404 returned error can't find the container with id 9664ce6804b382ec825630da5fbd42112167c597568f1b81e5390efa41b8540c Sep 16 14:07:12.755700 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.755681 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:07:12.756801 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:12.756782 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz"] Sep 16 14:07:13.048242 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:13.048168 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" event={"ID":"161d9559-3bfa-4713-92f5-1e5764240302","Type":"ContainerStarted","Data":"9664ce6804b382ec825630da5fbd42112167c597568f1b81e5390efa41b8540c"} Sep 16 14:07:17.076917 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:17.076878 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" event={"ID":"161d9559-3bfa-4713-92f5-1e5764240302","Type":"ContainerStarted","Data":"c52c5c6fbc7f99864e52a7f729e1272695c98e327f1c29e30da497d6464b2fcb"} Sep 16 14:07:17.077338 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:17.076942 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:17.109391 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:17.109344 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" podStartSLOduration=1.663030483 podStartE2EDuration="5.109328437s" totalImagesPullingTime="3.446297954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:17.107000958 +0000 UTC m=+1471.070359814" watchObservedRunningTime="2026-09-16 14:07:17.109328437 +0000 UTC m=+1471.072687319" Sep 16 14:07:23.088559 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:23.088523 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:07:44.064032 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.063989 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n"] Sep 16 14:07:44.069109 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.069084 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.071952 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.071926 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8a4447-serving-cert\"" Sep 16 14:07:44.095877 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.095852 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n"] Sep 16 14:07:44.156847 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.156814 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-maas-api-tls\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.157001 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.156854 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbl8\" (UniqueName: \"kubernetes.io/projected/6c3dfad0-e597-4331-a114-0efa555f91f2-kube-api-access-sdbl8\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.157001 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.156941 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c3dfad0-e597-4331-a114-0efa555f91f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.157001 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.156995 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-metrics-certs\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.258469 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.258431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-maas-api-tls\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.258656 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.258488 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbl8\" (UniqueName: \"kubernetes.io/projected/6c3dfad0-e597-4331-a114-0efa555f91f2-kube-api-access-sdbl8\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.258656 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.258569 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c3dfad0-e597-4331-a114-0efa555f91f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.258656 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.258639 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-metrics-certs\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.259249 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.259225 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c3dfad0-e597-4331-a114-0efa555f91f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.260982 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.260949 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-maas-api-tls\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.261124 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.261106 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-metrics-certs\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.270256 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.270233 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbl8\" (UniqueName: \"kubernetes.io/projected/6c3dfad0-e597-4331-a114-0efa555f91f2-kube-api-access-sdbl8\") pod \"maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.380495 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.380409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:44.671685 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:44.671639 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n"] Sep 16 14:07:44.674053 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:07:44.674021 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3dfad0_e597_4331_a114_0efa555f91f2.slice/crio-c3aedea5e6986f65226480b4721afc739538e67d81424416aa94aa27e14f3567 WatchSource:0}: Error finding container c3aedea5e6986f65226480b4721afc739538e67d81424416aa94aa27e14f3567: Status 404 returned error can't find the container with id c3aedea5e6986f65226480b4721afc739538e67d81424416aa94aa27e14f3567 Sep 16 14:07:45.251291 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.251255 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" event={"ID":"6c3dfad0-e597-4331-a114-0efa555f91f2","Type":"ContainerStarted","Data":"c3aedea5e6986f65226480b4721afc739538e67d81424416aa94aa27e14f3567"} Sep 16 14:07:45.487445 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.487410 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t"] Sep 16 14:07:45.487613 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.487596 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" podUID="6ae5bcca-86ee-43b0-b0d6-b09529a19152" containerName="maas-api" containerID="cri-o://82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900" gracePeriod=30 Sep 16 14:07:45.754975 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.754916 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 14:07:45.877386 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.877351 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-metrics-certs\") pod \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " Sep 16 14:07:45.877572 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.877408 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-maas-api-tls\") pod \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " Sep 16 14:07:45.877572 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.877469 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae5bcca-86ee-43b0-b0d6-b09529a19152-openshift-service-ca\") pod \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " Sep 16 14:07:45.877720 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.877693 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68j9m\" (UniqueName: \"kubernetes.io/projected/6ae5bcca-86ee-43b0-b0d6-b09529a19152-kube-api-access-68j9m\") pod \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\" (UID: \"6ae5bcca-86ee-43b0-b0d6-b09529a19152\") " Sep 16 14:07:45.877913 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.877864 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae5bcca-86ee-43b0-b0d6-b09529a19152-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ae5bcca-86ee-43b0-b0d6-b09529a19152" (UID: "6ae5bcca-86ee-43b0-b0d6-b09529a19152"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:07:45.878132 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.878105 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae5bcca-86ee-43b0-b0d6-b09529a19152-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:07:45.879706 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.879678 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6ae5bcca-86ee-43b0-b0d6-b09529a19152" (UID: "6ae5bcca-86ee-43b0-b0d6-b09529a19152"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:07:45.880078 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.880055 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae5bcca-86ee-43b0-b0d6-b09529a19152-kube-api-access-68j9m" (OuterVolumeSpecName: "kube-api-access-68j9m") pod "6ae5bcca-86ee-43b0-b0d6-b09529a19152" (UID: "6ae5bcca-86ee-43b0-b0d6-b09529a19152"). InnerVolumeSpecName "kube-api-access-68j9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:07:45.880164 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.880068 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ae5bcca-86ee-43b0-b0d6-b09529a19152" (UID: "6ae5bcca-86ee-43b0-b0d6-b09529a19152"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:07:45.979013 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.978976 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:07:45.979013 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.979010 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68j9m\" (UniqueName: \"kubernetes.io/projected/6ae5bcca-86ee-43b0-b0d6-b09529a19152-kube-api-access-68j9m\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:07:45.979246 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:45.979024 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae5bcca-86ee-43b0-b0d6-b09529a19152-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:07:46.258926 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.258826 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" event={"ID":"6c3dfad0-e597-4331-a114-0efa555f91f2","Type":"ContainerStarted","Data":"ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a"} Sep 16 14:07:46.259360 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.259023 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:07:46.260362 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.260331 2581 generic.go:356] "Generic (PLEG): container finished" podID="6ae5bcca-86ee-43b0-b0d6-b09529a19152" containerID="82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900" exitCode=0 Sep 16 14:07:46.260502 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.260391 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" Sep 16 14:07:46.260502 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.260414 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" event={"ID":"6ae5bcca-86ee-43b0-b0d6-b09529a19152","Type":"ContainerDied","Data":"82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900"} Sep 16 14:07:46.260502 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.260457 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" event={"ID":"6ae5bcca-86ee-43b0-b0d6-b09529a19152","Type":"ContainerDied","Data":"43cb0e64f6c1f6abe1bf03f8cf062958bca26d568722a625622066070314463d"} Sep 16 14:07:46.260502 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.260486 2581 scope.go:117] "RemoveContainer" containerID="82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900" Sep 16 14:07:46.275211 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.275192 2581 scope.go:117] "RemoveContainer" containerID="82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900" Sep 16 14:07:46.275493 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:07:46.275474 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900\": container with ID starting with 82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900 not found: ID does not exist" containerID="82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900" Sep 16 14:07:46.275557 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.275500 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900"} err="failed to get container status \"82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900\": rpc error: code = NotFound desc = could not find container \"82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900\": container with ID starting with 82e226230d28f2e0ad1c79bbffe2a8eebce5ce31049f58201d8433a1fe75d900 not found: ID does not exist" Sep 16 14:07:46.294082 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.294032 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" podStartSLOduration=0.979049767 podStartE2EDuration="2.294018411s" totalImagesPullingTime="1.314968644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:46.293526663 +0000 UTC m=+1500.256885489" watchObservedRunningTime="2026-09-16 14:07:46.294018411 +0000 UTC m=+1500.257377237" Sep 16 14:07:46.329945 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.329911 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t"] Sep 16 14:07:46.341091 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.341045 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t"] Sep 16 14:07:46.517073 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.516997 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ae5bcca-86ee-43b0-b0d6-b09529a19152" path="/var/lib/kubelet/pods/6ae5bcca-86ee-43b0-b0d6-b09529a19152/volumes" Sep 16 14:07:46.633031 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.633006 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:07:46.640126 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.640103 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:07:46.642694 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.642671 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 14:07:46.650266 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:46.650251 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 14:07:47.162155 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:47.162123 2581 scope.go:117] "RemoveContainer" containerID="50db27b94738fea29b0425aa5dac6937b77c654fbc724db9cc1a81461380abc4" Sep 16 14:07:48.722101 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:48.722057 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-19203d-79949f6dd9-tht9t" podUID="6ae5bcca-86ee-43b0-b0d6-b09529a19152" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.58:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 16 14:07:52.272640 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:07:52.272609 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:08:18.230098 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.230006 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz"] Sep 16 14:08:18.230679 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.230282 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" podUID="161d9559-3bfa-4713-92f5-1e5764240302" containerName="maas-api" containerID="cri-o://c52c5c6fbc7f99864e52a7f729e1272695c98e327f1c29e30da497d6464b2fcb" gracePeriod=30 Sep 16 14:08:18.450446 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.450414 2581 generic.go:356] "Generic (PLEG): container finished" podID="161d9559-3bfa-4713-92f5-1e5764240302" containerID="c52c5c6fbc7f99864e52a7f729e1272695c98e327f1c29e30da497d6464b2fcb" exitCode=0 Sep 16 14:08:18.450643 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.450498 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" event={"ID":"161d9559-3bfa-4713-92f5-1e5764240302","Type":"ContainerDied","Data":"c52c5c6fbc7f99864e52a7f729e1272695c98e327f1c29e30da497d6464b2fcb"} Sep 16 14:08:18.480415 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.480352 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:08:18.502551 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.502526 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-metrics-certs\") pod \"161d9559-3bfa-4713-92f5-1e5764240302\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " Sep 16 14:08:18.502694 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.502583 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/161d9559-3bfa-4713-92f5-1e5764240302-openshift-service-ca\") pod \"161d9559-3bfa-4713-92f5-1e5764240302\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " Sep 16 14:08:18.502694 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.502646 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-maas-api-tls\") pod \"161d9559-3bfa-4713-92f5-1e5764240302\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " Sep 16 14:08:18.502812 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.502719 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn7lj\" (UniqueName: \"kubernetes.io/projected/161d9559-3bfa-4713-92f5-1e5764240302-kube-api-access-hn7lj\") pod \"161d9559-3bfa-4713-92f5-1e5764240302\" (UID: \"161d9559-3bfa-4713-92f5-1e5764240302\") " Sep 16 14:08:18.503663 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.503615 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161d9559-3bfa-4713-92f5-1e5764240302-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "161d9559-3bfa-4713-92f5-1e5764240302" (UID: "161d9559-3bfa-4713-92f5-1e5764240302"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:08:18.505732 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.505686 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161d9559-3bfa-4713-92f5-1e5764240302-kube-api-access-hn7lj" (OuterVolumeSpecName: "kube-api-access-hn7lj") pod "161d9559-3bfa-4713-92f5-1e5764240302" (UID: "161d9559-3bfa-4713-92f5-1e5764240302"). InnerVolumeSpecName "kube-api-access-hn7lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:08:18.505882 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.505862 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "161d9559-3bfa-4713-92f5-1e5764240302" (UID: "161d9559-3bfa-4713-92f5-1e5764240302"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:08:18.505937 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.505894 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "161d9559-3bfa-4713-92f5-1e5764240302" (UID: "161d9559-3bfa-4713-92f5-1e5764240302"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:08:18.604365 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.604335 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/161d9559-3bfa-4713-92f5-1e5764240302-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:08:18.604365 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.604367 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:08:18.604532 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.604381 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn7lj\" (UniqueName: \"kubernetes.io/projected/161d9559-3bfa-4713-92f5-1e5764240302-kube-api-access-hn7lj\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:08:18.604532 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:18.604392 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/161d9559-3bfa-4713-92f5-1e5764240302-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:08:19.457868 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:19.457835 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" Sep 16 14:08:19.457868 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:19.457849 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz" event={"ID":"161d9559-3bfa-4713-92f5-1e5764240302","Type":"ContainerDied","Data":"9664ce6804b382ec825630da5fbd42112167c597568f1b81e5390efa41b8540c"} Sep 16 14:08:19.458368 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:19.457940 2581 scope.go:117] "RemoveContainer" containerID="c52c5c6fbc7f99864e52a7f729e1272695c98e327f1c29e30da497d6464b2fcb" Sep 16 14:08:19.484233 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:19.484208 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz"] Sep 16 14:08:19.496456 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:19.496434 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f75a3-6bc8b7f99f-rhbtz"] Sep 16 14:08:20.516071 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:20.516032 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="161d9559-3bfa-4713-92f5-1e5764240302" path="/var/lib/kubelet/pods/161d9559-3bfa-4713-92f5-1e5764240302/volumes" Sep 16 14:08:41.592971 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.592936 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n"] Sep 16 14:08:41.593474 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.593209 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" podUID="6c3dfad0-e597-4331-a114-0efa555f91f2" containerName="maas-api" containerID="cri-o://ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a" gracePeriod=30 Sep 16 14:08:41.851853 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.851789 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:08:41.918521 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.918486 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-metrics-certs\") pod \"6c3dfad0-e597-4331-a114-0efa555f91f2\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " Sep 16 14:08:41.918684 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.918533 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-maas-api-tls\") pod \"6c3dfad0-e597-4331-a114-0efa555f91f2\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " Sep 16 14:08:41.918684 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.918594 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdbl8\" (UniqueName: \"kubernetes.io/projected/6c3dfad0-e597-4331-a114-0efa555f91f2-kube-api-access-sdbl8\") pod \"6c3dfad0-e597-4331-a114-0efa555f91f2\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " Sep 16 14:08:41.918684 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.918640 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c3dfad0-e597-4331-a114-0efa555f91f2-openshift-service-ca\") pod \"6c3dfad0-e597-4331-a114-0efa555f91f2\" (UID: \"6c3dfad0-e597-4331-a114-0efa555f91f2\") " Sep 16 14:08:41.920657 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.920125 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3dfad0-e597-4331-a114-0efa555f91f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c3dfad0-e597-4331-a114-0efa555f91f2" (UID: "6c3dfad0-e597-4331-a114-0efa555f91f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:08:41.921497 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.921461 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c3dfad0-e597-4331-a114-0efa555f91f2" (UID: "6c3dfad0-e597-4331-a114-0efa555f91f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:08:41.926184 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.923550 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3dfad0-e597-4331-a114-0efa555f91f2-kube-api-access-sdbl8" (OuterVolumeSpecName: "kube-api-access-sdbl8") pod "6c3dfad0-e597-4331-a114-0efa555f91f2" (UID: "6c3dfad0-e597-4331-a114-0efa555f91f2"). InnerVolumeSpecName "kube-api-access-sdbl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:08:41.927291 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:41.926753 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c3dfad0-e597-4331-a114-0efa555f91f2" (UID: "6c3dfad0-e597-4331-a114-0efa555f91f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:08:42.020676 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.020639 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:08:42.020917 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.020902 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c3dfad0-e597-4331-a114-0efa555f91f2-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:08:42.021023 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.021010 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbl8\" (UniqueName: \"kubernetes.io/projected/6c3dfad0-e597-4331-a114-0efa555f91f2-kube-api-access-sdbl8\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:08:42.021118 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.021104 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c3dfad0-e597-4331-a114-0efa555f91f2-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:08:42.597746 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.597708 2581 generic.go:356] "Generic (PLEG): container finished" podID="6c3dfad0-e597-4331-a114-0efa555f91f2" containerID="ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a" exitCode=0 Sep 16 14:08:42.598179 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.597747 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" event={"ID":"6c3dfad0-e597-4331-a114-0efa555f91f2","Type":"ContainerDied","Data":"ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a"} Sep 16 14:08:42.598179 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.597763 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" Sep 16 14:08:42.598179 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.597804 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n" event={"ID":"6c3dfad0-e597-4331-a114-0efa555f91f2","Type":"ContainerDied","Data":"c3aedea5e6986f65226480b4721afc739538e67d81424416aa94aa27e14f3567"} Sep 16 14:08:42.598179 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.597834 2581 scope.go:117] "RemoveContainer" containerID="ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a" Sep 16 14:08:42.607493 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.607476 2581 scope.go:117] "RemoveContainer" containerID="ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a" Sep 16 14:08:42.607750 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:08:42.607730 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a\": container with ID starting with ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a not found: ID does not exist" containerID="ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a" Sep 16 14:08:42.607860 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.607761 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a"} err="failed to get container status \"ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a\": rpc error: code = NotFound desc = could not find container \"ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a\": container with ID starting with ea19c56f97205c611a415d1e170aee88f6fb727a4f93f87d9d0bad679fbe730a not found: ID does not exist" Sep 16 14:08:42.618082 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.618060 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n"] Sep 16 14:08:42.625650 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:42.625627 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a4447-569b754cf9-dsj7n"] Sep 16 14:08:44.516725 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:08:44.516695 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c3dfad0-e597-4331-a114-0efa555f91f2" path="/var/lib/kubelet/pods/6c3dfad0-e597-4331-a114-0efa555f91f2/volumes" Sep 16 14:09:00.065750 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.065717 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz"] Sep 16 14:09:00.066319 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.065977 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" podUID="3748a3db-5c2d-4303-85e3-f4e8400c3b2f" containerName="maas-api" containerID="cri-o://a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698" gracePeriod=30 Sep 16 14:09:00.332373 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.332313 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:09:00.384925 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.384886 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-metrics-certs\") pod \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " Sep 16 14:09:00.385113 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.384952 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-openshift-service-ca\") pod \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " Sep 16 14:09:00.385113 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.385063 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-maas-api-tls\") pod \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " Sep 16 14:09:00.385113 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.385089 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlvfp\" (UniqueName: \"kubernetes.io/projected/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-kube-api-access-hlvfp\") pod \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\" (UID: \"3748a3db-5c2d-4303-85e3-f4e8400c3b2f\") " Sep 16 14:09:00.385458 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.385414 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3748a3db-5c2d-4303-85e3-f4e8400c3b2f" (UID: "3748a3db-5c2d-4303-85e3-f4e8400c3b2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:00.387268 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.387222 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3748a3db-5c2d-4303-85e3-f4e8400c3b2f" (UID: "3748a3db-5c2d-4303-85e3-f4e8400c3b2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:09:00.387268 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.387242 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3748a3db-5c2d-4303-85e3-f4e8400c3b2f" (UID: "3748a3db-5c2d-4303-85e3-f4e8400c3b2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:09:00.387427 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.387342 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-kube-api-access-hlvfp" (OuterVolumeSpecName: "kube-api-access-hlvfp") pod "3748a3db-5c2d-4303-85e3-f4e8400c3b2f" (UID: "3748a3db-5c2d-4303-85e3-f4e8400c3b2f"). InnerVolumeSpecName "kube-api-access-hlvfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:00.486613 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.486572 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:09:00.486613 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.486604 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlvfp\" (UniqueName: \"kubernetes.io/projected/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-kube-api-access-hlvfp\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:09:00.486613 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.486614 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:09:00.486613 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.486622 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3748a3db-5c2d-4303-85e3-f4e8400c3b2f-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:09:00.710085 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.710048 2581 generic.go:356] "Generic (PLEG): container finished" podID="3748a3db-5c2d-4303-85e3-f4e8400c3b2f" containerID="a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698" exitCode=0 Sep 16 14:09:00.710270 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.710095 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" Sep 16 14:09:00.710270 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.710088 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" event={"ID":"3748a3db-5c2d-4303-85e3-f4e8400c3b2f","Type":"ContainerDied","Data":"a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698"} Sep 16 14:09:00.710270 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.710200 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz" event={"ID":"3748a3db-5c2d-4303-85e3-f4e8400c3b2f","Type":"ContainerDied","Data":"c861642ff4409a5b40f7492a8a23647673966cdfd793c79ec329c07a3a19361e"} Sep 16 14:09:00.710270 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.710232 2581 scope.go:117] "RemoveContainer" containerID="a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698" Sep 16 14:09:00.719916 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.719894 2581 scope.go:117] "RemoveContainer" containerID="a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698" Sep 16 14:09:00.720213 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:09:00.720192 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698\": container with ID starting with a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698 not found: ID does not exist" containerID="a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698" Sep 16 14:09:00.720272 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.720222 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698"} err="failed to get container status \"a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698\": rpc error: code = NotFound desc = could not find container \"a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698\": container with ID starting with a89d83026ed8c0e1cdc0483e13feb097d36e415afe8b59078ddd8e37d2a83698 not found: ID does not exist" Sep 16 14:09:00.735868 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.735845 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz"] Sep 16 14:09:00.740505 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:00.740485 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-740a25-54db9b46c8-xk9sz"] Sep 16 14:09:02.515918 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:02.515881 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3748a3db-5c2d-4303-85e3-f4e8400c3b2f" path="/var/lib/kubelet/pods/3748a3db-5c2d-4303-85e3-f4e8400c3b2f/volumes" Sep 16 14:09:18.363351 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.363313 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn"] Sep 16 14:09:18.363829 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.363591 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" podUID="8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb" containerName="maas-api" containerID="cri-o://af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c" gracePeriod=30 Sep 16 14:09:18.612291 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.612264 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:09:18.650786 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.650724 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-metrics-certs\") pod \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " Sep 16 14:09:18.650917 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.650795 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-maas-api-tls\") pod \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " Sep 16 14:09:18.650917 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.650851 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrd2n\" (UniqueName: \"kubernetes.io/projected/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-kube-api-access-xrd2n\") pod \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " Sep 16 14:09:18.650917 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.650868 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-openshift-service-ca\") pod \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\" (UID: \"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb\") " Sep 16 14:09:18.651346 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.651318 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb" (UID: "8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:09:18.652923 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.652824 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb" (UID: "8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:09:18.652923 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.652883 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-kube-api-access-xrd2n" (OuterVolumeSpecName: "kube-api-access-xrd2n") pod "8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb" (UID: "8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb"). InnerVolumeSpecName "kube-api-access-xrd2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:09:18.652923 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.652880 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb" (UID: "8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:09:18.751872 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.751836 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrd2n\" (UniqueName: \"kubernetes.io/projected/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-kube-api-access-xrd2n\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:09:18.752090 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.752063 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:09:18.752090 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.752089 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:09:18.752259 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.752102 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:09:18.820292 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.820253 2581 generic.go:356] "Generic (PLEG): container finished" podID="8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb" containerID="af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c" exitCode=0 Sep 16 14:09:18.820439 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.820351 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" Sep 16 14:09:18.820439 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.820388 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" event={"ID":"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb","Type":"ContainerDied","Data":"af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c"} Sep 16 14:09:18.820567 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.820447 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn" event={"ID":"8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb","Type":"ContainerDied","Data":"1f2026df2a311aa0c9cc8db44e0168a32e07f63386822bb32038ed2f4fc90157"} Sep 16 14:09:18.820567 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.820478 2581 scope.go:117] "RemoveContainer" containerID="af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c" Sep 16 14:09:18.833956 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.832498 2581 scope.go:117] "RemoveContainer" containerID="af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c" Sep 16 14:09:18.834361 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:09:18.834336 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c\": container with ID starting with af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c not found: ID does not exist" containerID="af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c" Sep 16 14:09:18.834463 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.834371 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c"} err="failed to get container status \"af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c\": rpc error: code = NotFound desc = could not find container \"af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c\": container with ID starting with af53457d9195dc7ff49a4faa2d368e134be1992e821371f954edc3c30611188c not found: ID does not exist" Sep 16 14:09:18.844411 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.844387 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn"] Sep 16 14:09:18.849103 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:18.849085 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9187f5-6bbf477d4d-ndxmn"] Sep 16 14:09:20.518159 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:09:20.518103 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb" path="/var/lib/kubelet/pods/8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb/volumes" Sep 16 14:11:10.342774 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.342699 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp"] Sep 16 14:11:10.343235 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343201 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3748a3db-5c2d-4303-85e3-f4e8400c3b2f" containerName="maas-api" Sep 16 14:11:10.343235 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343218 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3748a3db-5c2d-4303-85e3-f4e8400c3b2f" containerName="maas-api" Sep 16 14:11:10.343396 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343241 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c3dfad0-e597-4331-a114-0efa555f91f2" containerName="maas-api" Sep 16 14:11:10.343396 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343250 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c3dfad0-e597-4331-a114-0efa555f91f2" containerName="maas-api" Sep 16 14:11:10.343396 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343285 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae5bcca-86ee-43b0-b0d6-b09529a19152" containerName="maas-api" Sep 16 14:11:10.343396 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343297 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae5bcca-86ee-43b0-b0d6-b09529a19152" containerName="maas-api" Sep 16 14:11:10.343396 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343308 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb" containerName="maas-api" Sep 16 14:11:10.343396 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343316 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb" containerName="maas-api" Sep 16 14:11:10.343396 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343331 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161d9559-3bfa-4713-92f5-1e5764240302" containerName="maas-api" Sep 16 14:11:10.343396 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343342 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="161d9559-3bfa-4713-92f5-1e5764240302" containerName="maas-api" Sep 16 14:11:10.343648 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343444 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="161d9559-3bfa-4713-92f5-1e5764240302" containerName="maas-api" Sep 16 14:11:10.343648 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343457 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3748a3db-5c2d-4303-85e3-f4e8400c3b2f" containerName="maas-api" Sep 16 14:11:10.343648 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343467 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c3dfad0-e597-4331-a114-0efa555f91f2" containerName="maas-api" Sep 16 14:11:10.343648 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343477 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fd27dd5-c077-44ef-8fb6-7da49b3ae1eb" containerName="maas-api" Sep 16 14:11:10.343648 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.343490 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae5bcca-86ee-43b0-b0d6-b09529a19152" containerName="maas-api" Sep 16 14:11:10.346807 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.346788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.349455 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.349430 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-cabdc2-serving-cert\"" Sep 16 14:11:10.365526 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.365497 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp"] Sep 16 14:11:10.409690 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.409661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-metrics-certs\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.409809 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.409715 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbphp\" (UniqueName: \"kubernetes.io/projected/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-kube-api-access-bbphp\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.409809 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.409744 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.409809 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.409779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.510823 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.510792 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.510981 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.510837 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.510981 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.510882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-metrics-certs\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.510981 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.510915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbphp\" (UniqueName: \"kubernetes.io/projected/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-kube-api-access-bbphp\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.510981 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:11:10.510930 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-serving-cert: secret "maas-api-e2e-ipp-a-cabdc2-serving-cert" not found Sep 16 14:11:10.511172 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:11:10.511018 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-maas-api-tls podName:4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1 nodeName:}" failed. No retries permitted until 2026-09-16 14:11:11.010996567 +0000 UTC m=+1704.974355385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-maas-api-tls") pod "maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" (UID: "4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1") : secret "maas-api-e2e-ipp-a-cabdc2-serving-cert" not found Sep 16 14:11:10.511637 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.511609 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.513614 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.513594 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-metrics-certs\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:10.523216 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:10.523192 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbphp\" (UniqueName: \"kubernetes.io/projected/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-kube-api-access-bbphp\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:11.017341 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:11.017305 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:11.020044 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:11.020019 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:11.260452 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:11.260415 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:11.536947 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:11.536920 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp"] Sep 16 14:11:11.538885 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:11:11.538822 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffa9208_72b2_4408_a32d_9b4f2b8a5cd1.slice/crio-3c93d50cce60f5d33096090993d740233731cc000e20ee6556f5d05ca80cb699 WatchSource:0}: Error finding container 3c93d50cce60f5d33096090993d740233731cc000e20ee6556f5d05ca80cb699: Status 404 returned error can't find the container with id 3c93d50cce60f5d33096090993d740233731cc000e20ee6556f5d05ca80cb699 Sep 16 14:11:12.469381 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:12.469350 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" event={"ID":"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1","Type":"ContainerStarted","Data":"b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7"} Sep 16 14:11:12.469558 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:12.469398 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" event={"ID":"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1","Type":"ContainerStarted","Data":"3c93d50cce60f5d33096090993d740233731cc000e20ee6556f5d05ca80cb699"} Sep 16 14:11:12.469558 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:12.469498 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:12.491353 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:12.491311 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" podStartSLOduration=1.863644952 podStartE2EDuration="2.491298564s" totalImagesPullingTime="627.653612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:12.488402368 +0000 UTC m=+1706.451761195" watchObservedRunningTime="2026-09-16 14:11:12.491298564 +0000 UTC m=+1706.454657391" Sep 16 14:11:18.480164 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:18.480114 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:11:40.234234 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.234192 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l"] Sep 16 14:11:40.237554 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.237537 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.241081 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.241048 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9c8daa-serving-cert\"" Sep 16 14:11:40.251523 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.251503 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l"] Sep 16 14:11:40.288410 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.288381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.288563 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.288441 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjcd\" (UniqueName: \"kubernetes.io/projected/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-kube-api-access-2tjcd\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.288563 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.288532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-metrics-certs\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.288665 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.288574 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.389320 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.389283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjcd\" (UniqueName: \"kubernetes.io/projected/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-kube-api-access-2tjcd\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.389545 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.389364 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-metrics-certs\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.389545 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.389399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.389545 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.389437 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.389717 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:11:40.389556 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-serving-cert: secret "maas-api-e2e-ipp-b-9c8daa-serving-cert" not found Sep 16 14:11:40.389717 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:11:40.389641 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls podName:f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf nodeName:}" failed. No retries permitted until 2026-09-16 14:11:40.889623521 +0000 UTC m=+1734.852982326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls") pod "maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" (UID: "f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf") : secret "maas-api-e2e-ipp-b-9c8daa-serving-cert" not found Sep 16 14:11:40.389998 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.389974 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.391893 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.391875 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-metrics-certs\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.398920 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.398895 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjcd\" (UniqueName: \"kubernetes.io/projected/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-kube-api-access-2tjcd\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.893608 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:40.893574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:40.893796 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:11:40.893740 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-serving-cert: secret "maas-api-e2e-ipp-b-9c8daa-serving-cert" not found Sep 16 14:11:40.893870 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:11:40.893830 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls podName:f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf nodeName:}" failed. No retries permitted until 2026-09-16 14:11:41.89381152 +0000 UTC m=+1735.857170325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls") pod "maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" (UID: "f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf") : secret "maas-api-e2e-ipp-b-9c8daa-serving-cert" not found Sep 16 14:11:41.902357 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:41.902320 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:41.904713 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:41.904691 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:42.049682 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:42.049654 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:42.327578 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:42.327548 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l"] Sep 16 14:11:42.328814 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:11:42.328786 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cc8e7c_cd2c_4485_9e8e_27c0a3298caf.slice/crio-241b9cdf8671ebcec3cb34c0a219871858aae222ce0d9b80464541cb4699e0cb WatchSource:0}: Error finding container 241b9cdf8671ebcec3cb34c0a219871858aae222ce0d9b80464541cb4699e0cb: Status 404 returned error can't find the container with id 241b9cdf8671ebcec3cb34c0a219871858aae222ce0d9b80464541cb4699e0cb Sep 16 14:11:42.642312 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:42.642232 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" event={"ID":"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf","Type":"ContainerStarted","Data":"241b9cdf8671ebcec3cb34c0a219871858aae222ce0d9b80464541cb4699e0cb"} Sep 16 14:11:43.650913 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:43.650878 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" event={"ID":"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf","Type":"ContainerStarted","Data":"386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad"} Sep 16 14:11:43.651348 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:43.650969 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:11:43.671311 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:43.671266 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" podStartSLOduration=3.2940845100000002 podStartE2EDuration="3.671251276s" totalImagesPullingTime="377.166766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:43.669004864 +0000 UTC m=+1737.632363692" watchObservedRunningTime="2026-09-16 14:11:43.671251276 +0000 UTC m=+1737.634610104" Sep 16 14:11:49.662073 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:11:49.662042 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:12:07.402463 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:07.402427 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4kn2r_485872f2-06f0-40e9-99a9-f603cd536dd4/payload-processing/0.log" Sep 16 14:12:38.506792 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:38.506747 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4kn2r_485872f2-06f0-40e9-99a9-f603cd536dd4/payload-processing/0.log" Sep 16 14:12:45.741569 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.741536 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r"] Sep 16 14:12:45.751258 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.751228 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.754030 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.753987 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-29812a-serving-cert\"" Sep 16 14:12:45.754902 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.754878 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r"] Sep 16 14:12:45.892521 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.892484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svklp\" (UniqueName: \"kubernetes.io/projected/9727f61d-096e-4d39-944a-a6e13e993d7b-kube-api-access-svklp\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.892684 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.892549 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.892684 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.892595 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.892768 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.892689 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9727f61d-096e-4d39-944a-a6e13e993d7b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.993636 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.993565 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svklp\" (UniqueName: \"kubernetes.io/projected/9727f61d-096e-4d39-944a-a6e13e993d7b-kube-api-access-svklp\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.993636 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.993611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.993851 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.993640 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.993851 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.993689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9727f61d-096e-4d39-944a-a6e13e993d7b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.994373 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.994356 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9727f61d-096e-4d39-944a-a6e13e993d7b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.996347 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.996321 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:45.996441 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:45.996377 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:46.002381 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:46.002351 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svklp\" (UniqueName: \"kubernetes.io/projected/9727f61d-096e-4d39-944a-a6e13e993d7b-kube-api-access-svklp\") pod \"maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:46.065600 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:46.065574 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:46.353646 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:46.353451 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r"] Sep 16 14:12:46.356505 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:12:46.356474 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9727f61d_096e_4d39_944a_a6e13e993d7b.slice/crio-b06b82f9e30ac08bc77d38175ebfec0a3953a86c23be6aef2e5a8f76e0be85c8 WatchSource:0}: Error finding container b06b82f9e30ac08bc77d38175ebfec0a3953a86c23be6aef2e5a8f76e0be85c8: Status 404 returned error can't find the container with id b06b82f9e30ac08bc77d38175ebfec0a3953a86c23be6aef2e5a8f76e0be85c8 Sep 16 14:12:46.358070 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:46.358049 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:12:46.674847 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:46.674757 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:12:46.684881 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:46.684853 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 14:12:46.685033 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:46.685015 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:12:46.694742 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:46.694720 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 14:12:47.014802 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:47.014770 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" event={"ID":"9727f61d-096e-4d39-944a-a6e13e993d7b","Type":"ContainerStarted","Data":"1b556ae16bd000978ab41a5dbc3123ebfc3998899ca542836e4cbf769a385a2a"} Sep 16 14:12:47.015171 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:47.014823 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" event={"ID":"9727f61d-096e-4d39-944a-a6e13e993d7b","Type":"ContainerStarted","Data":"b06b82f9e30ac08bc77d38175ebfec0a3953a86c23be6aef2e5a8f76e0be85c8"} Sep 16 14:12:47.015171 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:47.014886 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:12:47.055437 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:47.055387 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" podStartSLOduration=1.548716843 podStartE2EDuration="2.055370801s" totalImagesPullingTime="506.653958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:12:47.052247947 +0000 UTC m=+1801.015606774" watchObservedRunningTime="2026-09-16 14:12:47.055370801 +0000 UTC m=+1801.018729629" Sep 16 14:12:53.026062 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:12:53.026031 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:13:24.010951 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.010917 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r"] Sep 16 14:13:24.011912 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.011215 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" podUID="9727f61d-096e-4d39-944a-a6e13e993d7b" containerName="maas-api" containerID="cri-o://1b556ae16bd000978ab41a5dbc3123ebfc3998899ca542836e4cbf769a385a2a" gracePeriod=30 Sep 16 14:13:24.235438 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.235398 2581 generic.go:356] "Generic (PLEG): container finished" podID="9727f61d-096e-4d39-944a-a6e13e993d7b" containerID="1b556ae16bd000978ab41a5dbc3123ebfc3998899ca542836e4cbf769a385a2a" exitCode=0 Sep 16 14:13:24.235610 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.235439 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" event={"ID":"9727f61d-096e-4d39-944a-a6e13e993d7b","Type":"ContainerDied","Data":"1b556ae16bd000978ab41a5dbc3123ebfc3998899ca542836e4cbf769a385a2a"} Sep 16 14:13:24.265269 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.265214 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:13:24.346462 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.346431 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-metrics-certs\") pod \"9727f61d-096e-4d39-944a-a6e13e993d7b\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " Sep 16 14:13:24.346633 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.346519 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9727f61d-096e-4d39-944a-a6e13e993d7b-openshift-service-ca\") pod \"9727f61d-096e-4d39-944a-a6e13e993d7b\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " Sep 16 14:13:24.346711 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.346684 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svklp\" (UniqueName: \"kubernetes.io/projected/9727f61d-096e-4d39-944a-a6e13e993d7b-kube-api-access-svklp\") pod \"9727f61d-096e-4d39-944a-a6e13e993d7b\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " Sep 16 14:13:24.346822 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.346798 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-maas-api-tls\") pod \"9727f61d-096e-4d39-944a-a6e13e993d7b\" (UID: \"9727f61d-096e-4d39-944a-a6e13e993d7b\") " Sep 16 14:13:24.346822 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.346818 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9727f61d-096e-4d39-944a-a6e13e993d7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9727f61d-096e-4d39-944a-a6e13e993d7b" (UID: "9727f61d-096e-4d39-944a-a6e13e993d7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:13:24.347213 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.347190 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9727f61d-096e-4d39-944a-a6e13e993d7b-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:13:24.348673 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.348641 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9727f61d-096e-4d39-944a-a6e13e993d7b" (UID: "9727f61d-096e-4d39-944a-a6e13e993d7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:13:24.348765 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.348727 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9727f61d-096e-4d39-944a-a6e13e993d7b" (UID: "9727f61d-096e-4d39-944a-a6e13e993d7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:13:24.348933 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.348908 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9727f61d-096e-4d39-944a-a6e13e993d7b-kube-api-access-svklp" (OuterVolumeSpecName: "kube-api-access-svklp") pod "9727f61d-096e-4d39-944a-a6e13e993d7b" (UID: "9727f61d-096e-4d39-944a-a6e13e993d7b"). InnerVolumeSpecName "kube-api-access-svklp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:13:24.448014 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.447974 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:13:24.448206 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.448023 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svklp\" (UniqueName: \"kubernetes.io/projected/9727f61d-096e-4d39-944a-a6e13e993d7b-kube-api-access-svklp\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:13:24.448206 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:24.448036 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9727f61d-096e-4d39-944a-a6e13e993d7b-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:13:25.242393 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:25.242365 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" Sep 16 14:13:25.242812 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:25.242405 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r" event={"ID":"9727f61d-096e-4d39-944a-a6e13e993d7b","Type":"ContainerDied","Data":"b06b82f9e30ac08bc77d38175ebfec0a3953a86c23be6aef2e5a8f76e0be85c8"} Sep 16 14:13:25.242812 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:25.242464 2581 scope.go:117] "RemoveContainer" containerID="1b556ae16bd000978ab41a5dbc3123ebfc3998899ca542836e4cbf769a385a2a" Sep 16 14:13:25.270264 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:25.270228 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r"] Sep 16 14:13:25.272685 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:25.272665 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29812a-6658765c77-xkn8r"] Sep 16 14:13:26.516182 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:26.516135 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9727f61d-096e-4d39-944a-a6e13e993d7b" path="/var/lib/kubelet/pods/9727f61d-096e-4d39-944a-a6e13e993d7b/volumes" Sep 16 14:13:50.686598 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.686559 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp"] Sep 16 14:13:50.687092 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.686817 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" podUID="4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1" containerName="maas-api" containerID="cri-o://b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7" gracePeriod=30 Sep 16 14:13:50.947576 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.947513 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:13:50.997074 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.997046 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-maas-api-tls\") pod \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " Sep 16 14:13:50.997250 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.997101 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-metrics-certs\") pod \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " Sep 16 14:13:50.997250 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.997211 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-openshift-service-ca\") pod \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " Sep 16 14:13:50.997333 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.997280 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbphp\" (UniqueName: \"kubernetes.io/projected/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-kube-api-access-bbphp\") pod \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\" (UID: \"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1\") " Sep 16 14:13:50.997672 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.997636 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1" (UID: "4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:13:50.999242 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.999216 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1" (UID: "4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:13:50.999356 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.999257 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1" (UID: "4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:13:50.999356 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:50.999324 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-kube-api-access-bbphp" (OuterVolumeSpecName: "kube-api-access-bbphp") pod "4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1" (UID: "4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1"). InnerVolumeSpecName "kube-api-access-bbphp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:13:51.098844 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.098803 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbphp\" (UniqueName: \"kubernetes.io/projected/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-kube-api-access-bbphp\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:13:51.098844 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.098844 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:13:51.099034 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.098859 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:13:51.099034 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.098873 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:13:51.397887 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.397851 2581 generic.go:356] "Generic (PLEG): container finished" podID="4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1" containerID="b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7" exitCode=0 Sep 16 14:13:51.398070 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.397898 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" Sep 16 14:13:51.398070 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.397942 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" event={"ID":"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1","Type":"ContainerDied","Data":"b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7"} Sep 16 14:13:51.398070 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.397995 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp" event={"ID":"4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1","Type":"ContainerDied","Data":"3c93d50cce60f5d33096090993d740233731cc000e20ee6556f5d05ca80cb699"} Sep 16 14:13:51.398070 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.398024 2581 scope.go:117] "RemoveContainer" containerID="b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7" Sep 16 14:13:51.407613 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.407597 2581 scope.go:117] "RemoveContainer" containerID="b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7" Sep 16 14:13:51.407881 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:13:51.407864 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7\": container with ID starting with b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7 not found: ID does not exist" containerID="b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7" Sep 16 14:13:51.407978 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.407892 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7"} err="failed to get container status \"b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7\": rpc error: code = NotFound desc = could not find container \"b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7\": container with ID starting with b55364314a4ca4eca5e38af6d225830c505f3ec17db96e33ebb4dea284c4b3b7 not found: ID does not exist" Sep 16 14:13:51.422021 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.421995 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp"] Sep 16 14:13:51.426343 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:51.426322 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cabdc2-d746f8cb6-fmmfp"] Sep 16 14:13:52.515949 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:13:52.515917 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1" path="/var/lib/kubelet/pods/4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1/volumes" Sep 16 14:14:57.500410 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.500378 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l"] Sep 16 14:14:57.500834 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.500627 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" podUID="f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf" containerName="maas-api" containerID="cri-o://386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad" gracePeriod=30 Sep 16 14:14:57.750313 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.750279 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:14:57.777676 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.777202 2581 generic.go:356] "Generic (PLEG): container finished" podID="f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf" containerID="386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad" exitCode=0 Sep 16 14:14:57.777676 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.777266 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" event={"ID":"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf","Type":"ContainerDied","Data":"386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad"} Sep 16 14:14:57.777676 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.777328 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" event={"ID":"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf","Type":"ContainerDied","Data":"241b9cdf8671ebcec3cb34c0a219871858aae222ce0d9b80464541cb4699e0cb"} Sep 16 14:14:57.777676 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.777273 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l" Sep 16 14:14:57.777676 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.777356 2581 scope.go:117] "RemoveContainer" containerID="386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad" Sep 16 14:14:57.790043 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.790023 2581 scope.go:117] "RemoveContainer" containerID="386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad" Sep 16 14:14:57.790331 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:14:57.790312 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad\": container with ID starting with 386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad not found: ID does not exist" containerID="386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad" Sep 16 14:14:57.790409 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.790339 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad"} err="failed to get container status \"386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad\": rpc error: code = NotFound desc = could not find container \"386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad\": container with ID starting with 386944470cba36ab3b99025a6b996edbc2565f70d4e97fccfc564c7805cf75ad not found: ID does not exist" Sep 16 14:14:57.898660 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.898622 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-metrics-certs\") pod \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " Sep 16 14:14:57.898660 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.898660 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls\") pod \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " Sep 16 14:14:57.898912 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.898742 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-openshift-service-ca\") pod \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " Sep 16 14:14:57.898912 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.898822 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tjcd\" (UniqueName: \"kubernetes.io/projected/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-kube-api-access-2tjcd\") pod \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\" (UID: \"f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf\") " Sep 16 14:14:57.899438 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.899154 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf" (UID: "f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:14:57.899438 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.899239 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:14:57.902395 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.902366 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf" (UID: "f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:14:57.902594 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.902496 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-kube-api-access-2tjcd" (OuterVolumeSpecName: "kube-api-access-2tjcd") pod "f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf" (UID: "f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf"). InnerVolumeSpecName "kube-api-access-2tjcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:14:57.903393 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.903365 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf" (UID: "f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:14:57.999838 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.999799 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:14:57.999838 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.999827 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:14:57.999838 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:57.999836 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tjcd\" (UniqueName: \"kubernetes.io/projected/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf-kube-api-access-2tjcd\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:14:58.106256 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:58.106222 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l"] Sep 16 14:14:58.114446 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:58.114417 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c8daa-7dd64767b-mrf7l"] Sep 16 14:14:58.515314 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:14:58.515282 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf" path="/var/lib/kubelet/pods/f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf/volumes" Sep 16 14:15:15.933531 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:15.933499 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj"] Sep 16 14:15:15.934046 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:15.933689 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" podUID="8dfc3bc4-d5f2-470e-86d7-5b41b39ef526" containerName="maas-api" containerID="cri-o://c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0" gracePeriod=30 Sep 16 14:15:16.179533 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.179505 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:15:16.279321 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.279278 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-metrics-certs\") pod \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " Sep 16 14:15:16.279499 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.279378 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls\") pod \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " Sep 16 14:15:16.279499 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.279427 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-openshift-service-ca\") pod \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " Sep 16 14:15:16.279499 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.279454 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8qmz\" (UniqueName: \"kubernetes.io/projected/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-kube-api-access-w8qmz\") pod \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\" (UID: \"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526\") " Sep 16 14:15:16.279853 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.279822 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8dfc3bc4-d5f2-470e-86d7-5b41b39ef526" (UID: "8dfc3bc4-d5f2-470e-86d7-5b41b39ef526"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:15:16.281791 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.281756 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8dfc3bc4-d5f2-470e-86d7-5b41b39ef526" (UID: "8dfc3bc4-d5f2-470e-86d7-5b41b39ef526"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:15:16.281897 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.281871 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8dfc3bc4-d5f2-470e-86d7-5b41b39ef526" (UID: "8dfc3bc4-d5f2-470e-86d7-5b41b39ef526"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:15:16.282010 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.281994 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-kube-api-access-w8qmz" (OuterVolumeSpecName: "kube-api-access-w8qmz") pod "8dfc3bc4-d5f2-470e-86d7-5b41b39ef526" (UID: "8dfc3bc4-d5f2-470e-86d7-5b41b39ef526"). InnerVolumeSpecName "kube-api-access-w8qmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:16.380866 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.380825 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:16.380866 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.380866 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:16.381114 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.380881 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8qmz\" (UniqueName: \"kubernetes.io/projected/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-kube-api-access-w8qmz\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:16.381114 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.380895 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:16.890535 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.890495 2581 generic.go:356] "Generic (PLEG): container finished" podID="8dfc3bc4-d5f2-470e-86d7-5b41b39ef526" containerID="c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0" exitCode=0 Sep 16 14:15:16.890737 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.890544 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" Sep 16 14:15:16.890737 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.890584 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" event={"ID":"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526","Type":"ContainerDied","Data":"c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0"} Sep 16 14:15:16.890737 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.890654 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj" event={"ID":"8dfc3bc4-d5f2-470e-86d7-5b41b39ef526","Type":"ContainerDied","Data":"2b9a1bb45eb3d86e251154e866827920058e6d73ed0a356aec75db16bc36d6ea"} Sep 16 14:15:16.890737 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.890686 2581 scope.go:117] "RemoveContainer" containerID="c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0" Sep 16 14:15:16.902432 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.902392 2581 scope.go:117] "RemoveContainer" containerID="c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0" Sep 16 14:15:16.902983 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:15:16.902951 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0\": container with ID starting with c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0 not found: ID does not exist" containerID="c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0" Sep 16 14:15:16.903085 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.902991 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0"} err="failed to get container status \"c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0\": rpc error: code = NotFound desc = could not find container \"c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0\": container with ID starting with c4bd0a54abde7ee4e975c4ec602ce0b7e1a82ec48b163a8def7bd0deabeaeeb0 not found: ID does not exist" Sep 16 14:15:16.920986 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.920959 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj"] Sep 16 14:15:16.927427 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:16.927405 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-46224d-846b78cb44-j2vwj"] Sep 16 14:15:18.522201 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:18.522166 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dfc3bc4-d5f2-470e-86d7-5b41b39ef526" path="/var/lib/kubelet/pods/8dfc3bc4-d5f2-470e-86d7-5b41b39ef526/volumes" Sep 16 14:15:34.778055 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:34.778022 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs"] Sep 16 14:15:34.778565 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:34.778275 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" podUID="9df2fc3d-cc16-42e4-9d81-5b436fe22fa4" containerName="maas-api" containerID="cri-o://405d2933155a74f53f9f7a1ece6ce35ef821a404d1dec0103c07f8e3261cc4b3" gracePeriod=30 Sep 16 14:15:34.996320 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:34.996281 2581 generic.go:356] "Generic (PLEG): container finished" podID="9df2fc3d-cc16-42e4-9d81-5b436fe22fa4" containerID="405d2933155a74f53f9f7a1ece6ce35ef821a404d1dec0103c07f8e3261cc4b3" exitCode=0 Sep 16 14:15:34.996484 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:34.996359 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" event={"ID":"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4","Type":"ContainerDied","Data":"405d2933155a74f53f9f7a1ece6ce35ef821a404d1dec0103c07f8e3261cc4b3"} Sep 16 14:15:35.031766 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.031707 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:15:35.146544 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.146513 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-metrics-certs\") pod \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " Sep 16 14:15:35.146732 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.146583 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-maas-api-tls\") pod \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " Sep 16 14:15:35.146732 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.146640 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5dkx\" (UniqueName: \"kubernetes.io/projected/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-kube-api-access-w5dkx\") pod \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " Sep 16 14:15:35.146732 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.146686 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-openshift-service-ca\") pod \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\" (UID: \"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4\") " Sep 16 14:15:35.147180 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.147128 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9df2fc3d-cc16-42e4-9d81-5b436fe22fa4" (UID: "9df2fc3d-cc16-42e4-9d81-5b436fe22fa4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:15:35.148691 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.148662 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9df2fc3d-cc16-42e4-9d81-5b436fe22fa4" (UID: "9df2fc3d-cc16-42e4-9d81-5b436fe22fa4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:15:35.148798 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.148707 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9df2fc3d-cc16-42e4-9d81-5b436fe22fa4" (UID: "9df2fc3d-cc16-42e4-9d81-5b436fe22fa4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:15:35.148858 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.148795 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-kube-api-access-w5dkx" (OuterVolumeSpecName: "kube-api-access-w5dkx") pod "9df2fc3d-cc16-42e4-9d81-5b436fe22fa4" (UID: "9df2fc3d-cc16-42e4-9d81-5b436fe22fa4"). InnerVolumeSpecName "kube-api-access-w5dkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:35.248115 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.248081 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5dkx\" (UniqueName: \"kubernetes.io/projected/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-kube-api-access-w5dkx\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:35.248115 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.248110 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:35.248115 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.248120 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:35.248349 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:35.248128 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:36.003609 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:36.003581 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" Sep 16 14:15:36.004056 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:36.003623 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs" event={"ID":"9df2fc3d-cc16-42e4-9d81-5b436fe22fa4","Type":"ContainerDied","Data":"9700aed1a8942582eae8c4924220c41ecf4eff6f47cea5047ecd7cd55ef55fbd"} Sep 16 14:15:36.004056 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:36.003689 2581 scope.go:117] "RemoveContainer" containerID="405d2933155a74f53f9f7a1ece6ce35ef821a404d1dec0103c07f8e3261cc4b3" Sep 16 14:15:36.034859 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:36.034836 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs"] Sep 16 14:15:36.039706 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:36.039685 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b4884e-74687d8f54-lscfs"] Sep 16 14:15:36.515989 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:36.515951 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df2fc3d-cc16-42e4-9d81-5b436fe22fa4" path="/var/lib/kubelet/pods/9df2fc3d-cc16-42e4-9d81-5b436fe22fa4/volumes" Sep 16 14:15:44.944345 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:44.944260 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-k7vp7"] Sep 16 14:15:44.944886 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:44.944462 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" podUID="b99bc9d3-c7cd-4e4d-84b5-516314915be2" containerName="manager" containerID="cri-o://73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0" gracePeriod=30 Sep 16 14:15:45.193725 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:45.193701 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 14:15:45.335296 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:45.335263 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqnnl\" (UniqueName: \"kubernetes.io/projected/b99bc9d3-c7cd-4e4d-84b5-516314915be2-kube-api-access-fqnnl\") pod \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " Sep 16 14:15:45.335507 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:45.335333 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-metrics-certs\") pod \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " Sep 16 14:15:45.335507 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:45.335398 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-webhook-cert\") pod \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\" (UID: \"b99bc9d3-c7cd-4e4d-84b5-516314915be2\") " Sep 16 14:15:45.337385 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:45.337356 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b99bc9d3-c7cd-4e4d-84b5-516314915be2" (UID: "b99bc9d3-c7cd-4e4d-84b5-516314915be2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:15:45.337510 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:45.337409 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99bc9d3-c7cd-4e4d-84b5-516314915be2-kube-api-access-fqnnl" (OuterVolumeSpecName: "kube-api-access-fqnnl") pod "b99bc9d3-c7cd-4e4d-84b5-516314915be2" (UID: "b99bc9d3-c7cd-4e4d-84b5-516314915be2"). InnerVolumeSpecName "kube-api-access-fqnnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:15:45.337510 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:45.337412 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b99bc9d3-c7cd-4e4d-84b5-516314915be2" (UID: "b99bc9d3-c7cd-4e4d-84b5-516314915be2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:15:45.436296 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:45.436257 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqnnl\" (UniqueName: \"kubernetes.io/projected/b99bc9d3-c7cd-4e4d-84b5-516314915be2-kube-api-access-fqnnl\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:45.436296 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:45.436291 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:45.436296 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:45.436300 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b99bc9d3-c7cd-4e4d-84b5-516314915be2-webhook-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:15:46.063516 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:46.063462 2581 generic.go:356] "Generic (PLEG): container finished" podID="b99bc9d3-c7cd-4e4d-84b5-516314915be2" containerID="73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0" exitCode=0 Sep 16 14:15:46.064041 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:46.063528 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" Sep 16 14:15:46.064041 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:46.063539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" event={"ID":"b99bc9d3-c7cd-4e4d-84b5-516314915be2","Type":"ContainerDied","Data":"73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0"} Sep 16 14:15:46.064041 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:46.063617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d96bd79db-k7vp7" event={"ID":"b99bc9d3-c7cd-4e4d-84b5-516314915be2","Type":"ContainerDied","Data":"97f53c5fa7ae4d15af1c646f0ae33c073a6af07de885be4628337ec0722397f3"} Sep 16 14:15:46.064041 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:46.063639 2581 scope.go:117] "RemoveContainer" containerID="73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0" Sep 16 14:15:46.074730 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:46.074714 2581 scope.go:117] "RemoveContainer" containerID="73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0" Sep 16 14:15:46.075062 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:15:46.075041 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0\": container with ID starting with 73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0 not found: ID does not exist" containerID="73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0" Sep 16 14:15:46.075190 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:46.075069 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0"} err="failed to get container status \"73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0\": rpc error: code = NotFound desc = could not find container \"73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0\": container with ID starting with 73b64ee41097d2f30e90a3a5853009a2f2b3b655e3df1c7bdce25502bd7bceb0 not found: ID does not exist" Sep 16 14:15:46.093768 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:46.093739 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-k7vp7"] Sep 16 14:15:46.117342 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:46.117321 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-k7vp7"] Sep 16 14:15:46.516236 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:15:46.516202 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b99bc9d3-c7cd-4e4d-84b5-516314915be2" path="/var/lib/kubelet/pods/b99bc9d3-c7cd-4e4d-84b5-516314915be2/volumes" Sep 16 14:16:46.940301 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940267 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d96bd79db-fcxrm"] Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940626 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940637 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940652 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dfc3bc4-d5f2-470e-86d7-5b41b39ef526" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940659 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dfc3bc4-d5f2-470e-86d7-5b41b39ef526" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940671 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940676 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940687 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df2fc3d-cc16-42e4-9d81-5b436fe22fa4" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940693 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df2fc3d-cc16-42e4-9d81-5b436fe22fa4" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940698 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9727f61d-096e-4d39-944a-a6e13e993d7b" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940703 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9727f61d-096e-4d39-944a-a6e13e993d7b" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940712 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b99bc9d3-c7cd-4e4d-84b5-516314915be2" containerName="manager" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940717 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b99bc9d3-c7cd-4e4d-84b5-516314915be2" containerName="manager" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940780 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9727f61d-096e-4d39-944a-a6e13e993d7b" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940788 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dfc3bc4-d5f2-470e-86d7-5b41b39ef526" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940795 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df2fc3d-cc16-42e4-9d81-5b436fe22fa4" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940799 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ffa9208-72b2-4408-a32d-9b4f2b8a5cd1" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940805 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3cc8e7c-cd2c-4485-9e8e-27c0a3298caf" containerName="maas-api" Sep 16 14:16:46.942747 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.940812 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b99bc9d3-c7cd-4e4d-84b5-516314915be2" containerName="manager" Sep 16 14:16:46.943632 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.943616 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:46.951305 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.951282 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-brhwx\"" Sep 16 14:16:46.951416 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.951326 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 14:16:46.951416 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.951347 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 14:16:46.951416 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.951396 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 14:16:46.961002 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.960975 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-fcxrm"] Sep 16 14:16:46.979585 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.979540 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-webhook-cert\") pod \"maas-controller-7d96bd79db-fcxrm\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:46.979702 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.979628 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v652v\" (UniqueName: \"kubernetes.io/projected/8a126064-95d0-44ec-abd8-893e4a0b1e3e-kube-api-access-v652v\") pod \"maas-controller-7d96bd79db-fcxrm\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:46.979702 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:46.979647 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-metrics-certs\") pod \"maas-controller-7d96bd79db-fcxrm\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:47.080561 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:47.080525 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-webhook-cert\") pod \"maas-controller-7d96bd79db-fcxrm\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:47.080731 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:47.080591 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v652v\" (UniqueName: \"kubernetes.io/projected/8a126064-95d0-44ec-abd8-893e4a0b1e3e-kube-api-access-v652v\") pod \"maas-controller-7d96bd79db-fcxrm\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:47.080731 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:47.080610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-metrics-certs\") pod \"maas-controller-7d96bd79db-fcxrm\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:47.083093 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:47.083069 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-metrics-certs\") pod \"maas-controller-7d96bd79db-fcxrm\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:47.083093 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:47.083090 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-webhook-cert\") pod \"maas-controller-7d96bd79db-fcxrm\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:47.096091 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:47.096066 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v652v\" (UniqueName: \"kubernetes.io/projected/8a126064-95d0-44ec-abd8-893e4a0b1e3e-kube-api-access-v652v\") pod \"maas-controller-7d96bd79db-fcxrm\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:47.254094 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:47.254064 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:47.528182 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:47.528159 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-fcxrm"] Sep 16 14:16:47.530284 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:16:47.530257 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a126064_95d0_44ec_abd8_893e4a0b1e3e.slice/crio-bba5cba9e0f4664090e937bca200fad6b2228e3b818ca85a7263062b193c8cf3 WatchSource:0}: Error finding container bba5cba9e0f4664090e937bca200fad6b2228e3b818ca85a7263062b193c8cf3: Status 404 returned error can't find the container with id bba5cba9e0f4664090e937bca200fad6b2228e3b818ca85a7263062b193c8cf3 Sep 16 14:16:48.408705 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:48.408668 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" event={"ID":"8a126064-95d0-44ec-abd8-893e4a0b1e3e","Type":"ContainerStarted","Data":"bba5cba9e0f4664090e937bca200fad6b2228e3b818ca85a7263062b193c8cf3"} Sep 16 14:16:49.416610 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:49.416575 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" event={"ID":"8a126064-95d0-44ec-abd8-893e4a0b1e3e","Type":"ContainerStarted","Data":"c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f"} Sep 16 14:16:49.417086 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:49.416687 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:16:49.440268 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:16:49.440220 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" podStartSLOduration=2.546271965 podStartE2EDuration="3.440207169s" totalImagesPullingTime="893.935204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:49.438086932 +0000 UTC m=+2043.401445789" watchObservedRunningTime="2026-09-16 14:16:49.440207169 +0000 UTC m=+2043.403565996" Sep 16 14:17:00.425351 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:00.425317 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:17:07.661117 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:07.661071 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-fcxrm"] Sep 16 14:17:07.661739 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:07.661282 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" podUID="8a126064-95d0-44ec-abd8-893e4a0b1e3e" containerName="manager" containerID="cri-o://c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f" gracePeriod=30 Sep 16 14:17:07.897317 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:07.897293 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:17:08.068196 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.068117 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-webhook-cert\") pod \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " Sep 16 14:17:08.068196 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.068193 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-metrics-certs\") pod \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " Sep 16 14:17:08.068421 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.068312 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v652v\" (UniqueName: \"kubernetes.io/projected/8a126064-95d0-44ec-abd8-893e4a0b1e3e-kube-api-access-v652v\") pod \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\" (UID: \"8a126064-95d0-44ec-abd8-893e4a0b1e3e\") " Sep 16 14:17:08.070166 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.070107 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8a126064-95d0-44ec-abd8-893e4a0b1e3e" (UID: "8a126064-95d0-44ec-abd8-893e4a0b1e3e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:17:08.070283 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.070191 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a126064-95d0-44ec-abd8-893e4a0b1e3e" (UID: "8a126064-95d0-44ec-abd8-893e4a0b1e3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:17:08.070283 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.070221 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a126064-95d0-44ec-abd8-893e4a0b1e3e-kube-api-access-v652v" (OuterVolumeSpecName: "kube-api-access-v652v") pod "8a126064-95d0-44ec-abd8-893e4a0b1e3e" (UID: "8a126064-95d0-44ec-abd8-893e4a0b1e3e"). InnerVolumeSpecName "kube-api-access-v652v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:17:08.169543 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.169512 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v652v\" (UniqueName: \"kubernetes.io/projected/8a126064-95d0-44ec-abd8-893e4a0b1e3e-kube-api-access-v652v\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:17:08.169543 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.169537 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-webhook-cert\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:17:08.169543 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.169547 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a126064-95d0-44ec-abd8-893e4a0b1e3e-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:17:08.528563 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.528538 2581 generic.go:356] "Generic (PLEG): container finished" podID="8a126064-95d0-44ec-abd8-893e4a0b1e3e" containerID="c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f" exitCode=0 Sep 16 14:17:08.528710 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.528590 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" Sep 16 14:17:08.528710 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.528596 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" event={"ID":"8a126064-95d0-44ec-abd8-893e4a0b1e3e","Type":"ContainerDied","Data":"c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f"} Sep 16 14:17:08.528710 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.528707 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d96bd79db-fcxrm" event={"ID":"8a126064-95d0-44ec-abd8-893e4a0b1e3e","Type":"ContainerDied","Data":"bba5cba9e0f4664090e937bca200fad6b2228e3b818ca85a7263062b193c8cf3"} Sep 16 14:17:08.528867 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.528730 2581 scope.go:117] "RemoveContainer" containerID="c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f" Sep 16 14:17:08.539310 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.539286 2581 scope.go:117] "RemoveContainer" containerID="c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f" Sep 16 14:17:08.539593 ip-10-0-143-222 kubenswrapper[2581]: E0916 14:17:08.539570 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f\": container with ID starting with c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f not found: ID does not exist" containerID="c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f" Sep 16 14:17:08.539645 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.539600 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f"} err="failed to get container status \"c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f\": rpc error: code = NotFound desc = could not find container \"c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f\": container with ID starting with c5a5b8f9f1bef883a2cefa43547f3c7eb697bfb406266a463ac463d98959847f not found: ID does not exist" Sep 16 14:17:08.548675 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.548655 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-fcxrm"] Sep 16 14:17:08.556835 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:08.556809 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-fcxrm"] Sep 16 14:17:09.599118 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.599039 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d96bd79db-gd7cr"] Sep 16 14:17:09.602277 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.600037 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a126064-95d0-44ec-abd8-893e4a0b1e3e" containerName="manager" Sep 16 14:17:09.602277 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.600068 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a126064-95d0-44ec-abd8-893e4a0b1e3e" containerName="manager" Sep 16 14:17:09.602277 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.600405 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a126064-95d0-44ec-abd8-893e4a0b1e3e" containerName="manager" Sep 16 14:17:09.605961 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.605929 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:09.609406 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.609329 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 14:17:09.609406 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.609380 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 14:17:09.610245 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.610216 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 14:17:09.610417 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.610384 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-brhwx\"" Sep 16 14:17:09.611694 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.611672 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-gd7cr"] Sep 16 14:17:09.784717 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.784670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e75432b4-0b09-48dc-9bc1-7d69fdab6030-webhook-cert\") pod \"maas-controller-7d96bd79db-gd7cr\" (UID: \"e75432b4-0b09-48dc-9bc1-7d69fdab6030\") " pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:09.784922 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.784734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e75432b4-0b09-48dc-9bc1-7d69fdab6030-metrics-certs\") pod \"maas-controller-7d96bd79db-gd7cr\" (UID: \"e75432b4-0b09-48dc-9bc1-7d69fdab6030\") " pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:09.784922 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.784810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdt7\" (UniqueName: \"kubernetes.io/projected/e75432b4-0b09-48dc-9bc1-7d69fdab6030-kube-api-access-8bdt7\") pod \"maas-controller-7d96bd79db-gd7cr\" (UID: \"e75432b4-0b09-48dc-9bc1-7d69fdab6030\") " pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:09.886304 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.886208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e75432b4-0b09-48dc-9bc1-7d69fdab6030-webhook-cert\") pod \"maas-controller-7d96bd79db-gd7cr\" (UID: \"e75432b4-0b09-48dc-9bc1-7d69fdab6030\") " pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:09.886304 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.886270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e75432b4-0b09-48dc-9bc1-7d69fdab6030-metrics-certs\") pod \"maas-controller-7d96bd79db-gd7cr\" (UID: \"e75432b4-0b09-48dc-9bc1-7d69fdab6030\") " pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:09.886304 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.886300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdt7\" (UniqueName: \"kubernetes.io/projected/e75432b4-0b09-48dc-9bc1-7d69fdab6030-kube-api-access-8bdt7\") pod \"maas-controller-7d96bd79db-gd7cr\" (UID: \"e75432b4-0b09-48dc-9bc1-7d69fdab6030\") " pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:09.888895 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.888872 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e75432b4-0b09-48dc-9bc1-7d69fdab6030-webhook-cert\") pod \"maas-controller-7d96bd79db-gd7cr\" (UID: \"e75432b4-0b09-48dc-9bc1-7d69fdab6030\") " pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:09.889006 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.888872 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e75432b4-0b09-48dc-9bc1-7d69fdab6030-metrics-certs\") pod \"maas-controller-7d96bd79db-gd7cr\" (UID: \"e75432b4-0b09-48dc-9bc1-7d69fdab6030\") " pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:09.895581 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.895555 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdt7\" (UniqueName: \"kubernetes.io/projected/e75432b4-0b09-48dc-9bc1-7d69fdab6030-kube-api-access-8bdt7\") pod \"maas-controller-7d96bd79db-gd7cr\" (UID: \"e75432b4-0b09-48dc-9bc1-7d69fdab6030\") " pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:09.918362 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:09.918338 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:10.199001 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:10.198976 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d96bd79db-gd7cr"] Sep 16 14:17:10.200597 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:17:10.200565 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75432b4_0b09_48dc_9bc1_7d69fdab6030.slice/crio-ed8db91e4d26c047eba4c928d56325751e35f533df1b8363b7aa611bfb9f048f WatchSource:0}: Error finding container ed8db91e4d26c047eba4c928d56325751e35f533df1b8363b7aa611bfb9f048f: Status 404 returned error can't find the container with id ed8db91e4d26c047eba4c928d56325751e35f533df1b8363b7aa611bfb9f048f Sep 16 14:17:10.515233 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:10.515202 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a126064-95d0-44ec-abd8-893e4a0b1e3e" path="/var/lib/kubelet/pods/8a126064-95d0-44ec-abd8-893e4a0b1e3e/volumes" Sep 16 14:17:10.541279 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:10.541252 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d96bd79db-gd7cr" event={"ID":"e75432b4-0b09-48dc-9bc1-7d69fdab6030","Type":"ContainerStarted","Data":"ed8db91e4d26c047eba4c928d56325751e35f533df1b8363b7aa611bfb9f048f"} Sep 16 14:17:11.548452 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:11.548412 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d96bd79db-gd7cr" event={"ID":"e75432b4-0b09-48dc-9bc1-7d69fdab6030","Type":"ContainerStarted","Data":"ef181aca5d031d25cef87f322085b30c00eb28efda22d6f5533b04d190102b0e"} Sep 16 14:17:11.548853 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:11.548541 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:11.570124 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:11.570082 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d96bd79db-gd7cr" podStartSLOduration=2.166627124 podStartE2EDuration="2.570067639s" totalImagesPullingTime="403.440515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:17:11.567230231 +0000 UTC m=+2065.530589058" watchObservedRunningTime="2026-09-16 14:17:11.570067639 +0000 UTC m=+2065.533426516" Sep 16 14:17:22.556238 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:22.556207 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d96bd79db-gd7cr" Sep 16 14:17:46.722807 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:46.722780 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:17:46.732032 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:46.732009 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 14:17:46.732231 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:46.732214 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:17:46.740792 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:17:46.740769 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 14:21:35.519232 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.519193 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc"] Sep 16 14:21:35.523030 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.523007 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.541681 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.541651 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc"] Sep 16 14:21:35.668794 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.668759 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b898f392-814e-4c0f-a04d-33b936309c09-maas-api-tls\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.668956 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.668840 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b898f392-814e-4c0f-a04d-33b936309c09-openshift-service-ca\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.668956 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.668872 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4jx\" (UniqueName: \"kubernetes.io/projected/b898f392-814e-4c0f-a04d-33b936309c09-kube-api-access-dp4jx\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.668956 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.668901 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b898f392-814e-4c0f-a04d-33b936309c09-metrics-certs\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.769884 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.769810 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b898f392-814e-4c0f-a04d-33b936309c09-maas-api-tls\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.769884 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.769877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b898f392-814e-4c0f-a04d-33b936309c09-openshift-service-ca\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.770064 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.769904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4jx\" (UniqueName: \"kubernetes.io/projected/b898f392-814e-4c0f-a04d-33b936309c09-kube-api-access-dp4jx\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.770064 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.769927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b898f392-814e-4c0f-a04d-33b936309c09-metrics-certs\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.770778 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.770747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b898f392-814e-4c0f-a04d-33b936309c09-openshift-service-ca\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.772471 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.772446 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b898f392-814e-4c0f-a04d-33b936309c09-maas-api-tls\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.772583 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.772496 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b898f392-814e-4c0f-a04d-33b936309c09-metrics-certs\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.787331 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.787306 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4jx\" (UniqueName: \"kubernetes.io/projected/b898f392-814e-4c0f-a04d-33b936309c09-kube-api-access-dp4jx\") pod \"maas-api-fd9d6989-8wmzc\" (UID: \"b898f392-814e-4c0f-a04d-33b936309c09\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:35.835560 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:35.835533 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:36.125958 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:36.125930 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc"] Sep 16 14:21:36.127335 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:21:36.127304 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb898f392_814e_4c0f_a04d_33b936309c09.slice/crio-6af73396104179eb11fc5ee328af953493c4f03651165534ff2dbfaaa3f03d09 WatchSource:0}: Error finding container 6af73396104179eb11fc5ee328af953493c4f03651165534ff2dbfaaa3f03d09: Status 404 returned error can't find the container with id 6af73396104179eb11fc5ee328af953493c4f03651165534ff2dbfaaa3f03d09 Sep 16 14:21:36.128722 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:36.128701 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:21:37.052720 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:37.052684 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" event={"ID":"b898f392-814e-4c0f-a04d-33b936309c09","Type":"ContainerStarted","Data":"6af73396104179eb11fc5ee328af953493c4f03651165534ff2dbfaaa3f03d09"} Sep 16 14:21:38.060206 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:38.060160 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" event={"ID":"b898f392-814e-4c0f-a04d-33b936309c09","Type":"ContainerStarted","Data":"3564f7ed48c7ebed50c17ea0234af5a3d9055e10621181da4fda9886151b1a5d"} Sep 16 14:21:38.060656 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:38.060251 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:38.088940 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:38.088892 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" podStartSLOduration=1.9204776190000001 podStartE2EDuration="3.08887837s" totalImagesPullingTime="1.168400751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:21:38.088029532 +0000 UTC m=+2332.051388359" watchObservedRunningTime="2026-09-16 14:21:38.08887837 +0000 UTC m=+2332.052237197" Sep 16 14:21:44.072105 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:44.072069 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-fd9d6989-8wmzc" Sep 16 14:21:51.961773 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:51.961729 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2"] Sep 16 14:21:51.964265 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:51.961979 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" podUID="2e0461a8-b839-4c98-acf6-baed82905c86" containerName="maas-api" containerID="cri-o://8ae7af8c577fe5b0bd80a2be097237b24f022ec98d630db5f14a881d9d845ad3" gracePeriod=30 Sep 16 14:21:53.149266 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.149219 2581 generic.go:356] "Generic (PLEG): container finished" podID="2e0461a8-b839-4c98-acf6-baed82905c86" containerID="8ae7af8c577fe5b0bd80a2be097237b24f022ec98d630db5f14a881d9d845ad3" exitCode=0 Sep 16 14:21:53.149605 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.149287 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" event={"ID":"2e0461a8-b839-4c98-acf6-baed82905c86","Type":"ContainerDied","Data":"8ae7af8c577fe5b0bd80a2be097237b24f022ec98d630db5f14a881d9d845ad3"} Sep 16 14:21:53.209239 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.209216 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 14:21:53.222540 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.222520 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw9db\" (UniqueName: \"kubernetes.io/projected/2e0461a8-b839-4c98-acf6-baed82905c86-kube-api-access-cw9db\") pod \"2e0461a8-b839-4c98-acf6-baed82905c86\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " Sep 16 14:21:53.222628 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.222562 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-maas-api-tls\") pod \"2e0461a8-b839-4c98-acf6-baed82905c86\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " Sep 16 14:21:53.222628 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.222599 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0461a8-b839-4c98-acf6-baed82905c86-openshift-service-ca\") pod \"2e0461a8-b839-4c98-acf6-baed82905c86\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " Sep 16 14:21:53.222705 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.222636 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-metrics-certs\") pod \"2e0461a8-b839-4c98-acf6-baed82905c86\" (UID: \"2e0461a8-b839-4c98-acf6-baed82905c86\") " Sep 16 14:21:53.222982 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.222955 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0461a8-b839-4c98-acf6-baed82905c86-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e0461a8-b839-4c98-acf6-baed82905c86" (UID: "2e0461a8-b839-4c98-acf6-baed82905c86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:21:53.224657 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.224630 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e0461a8-b839-4c98-acf6-baed82905c86" (UID: "2e0461a8-b839-4c98-acf6-baed82905c86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:21:53.224759 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.224692 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e0461a8-b839-4c98-acf6-baed82905c86" (UID: "2e0461a8-b839-4c98-acf6-baed82905c86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:21:53.224822 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.224803 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0461a8-b839-4c98-acf6-baed82905c86-kube-api-access-cw9db" (OuterVolumeSpecName: "kube-api-access-cw9db") pod "2e0461a8-b839-4c98-acf6-baed82905c86" (UID: "2e0461a8-b839-4c98-acf6-baed82905c86"). InnerVolumeSpecName "kube-api-access-cw9db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:21:53.323636 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.323575 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0461a8-b839-4c98-acf6-baed82905c86-openshift-service-ca\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:21:53.323636 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.323600 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-metrics-certs\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:21:53.323636 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.323609 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw9db\" (UniqueName: \"kubernetes.io/projected/2e0461a8-b839-4c98-acf6-baed82905c86-kube-api-access-cw9db\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:21:53.323636 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:53.323618 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0461a8-b839-4c98-acf6-baed82905c86-maas-api-tls\") on node \"ip-10-0-143-222.ec2.internal\" DevicePath \"\"" Sep 16 14:21:54.157008 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:54.156978 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" Sep 16 14:21:54.157504 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:54.157018 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2" event={"ID":"2e0461a8-b839-4c98-acf6-baed82905c86","Type":"ContainerDied","Data":"39d6288c03139fd87e10ca4321e7522bf8e3ad9707462ce0a1adfd45fba48bd5"} Sep 16 14:21:54.157504 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:54.157076 2581 scope.go:117] "RemoveContainer" containerID="8ae7af8c577fe5b0bd80a2be097237b24f022ec98d630db5f14a881d9d845ad3" Sep 16 14:21:54.182822 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:54.182795 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2"] Sep 16 14:21:54.187632 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:54.187610 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-9h6d2"] Sep 16 14:21:54.516164 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:21:54.516114 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e0461a8-b839-4c98-acf6-baed82905c86" path="/var/lib/kubelet/pods/2e0461a8-b839-4c98-acf6-baed82905c86/volumes" Sep 16 14:22:12.302541 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:12.302510 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4gx77_4d68b82b-d7a5-44dc-a602-229ecfe490c7/manager/0.log" Sep 16 14:22:12.427639 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:12.427586 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d96bd79db-gd7cr_e75432b4-0b09-48dc-9bc1-7d69fdab6030/manager/0.log" Sep 16 14:22:13.033880 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:13.033853 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-wm9r4_382e3f4d-0714-4b70-8e27-d471c525de0e/copy-manifests/0.log" Sep 16 14:22:13.046229 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:13.046203 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-wm9r4_382e3f4d-0714-4b70-8e27-d471c525de0e/manager/0.log" Sep 16 14:22:13.487286 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:13.487254 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-fd9d6989-8wmzc_b898f392-814e-4c0f-a04d-33b936309c09/maas-api/0.log" Sep 16 14:22:13.830344 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:13.830267 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-2pwzd_5ed7672c-8453-4190-9f24-19f007282c60/postgres/0.log" Sep 16 14:22:15.328351 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:15.328316 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-nvld6_b83b14bf-169c-4d36-8a6a-e3098c9c0841/manager/0.log" Sep 16 14:22:15.806722 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:15.806693 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vxrxf_dff9285c-d48d-437a-8a49-97faee1752d0/discovery/0.log" Sep 16 14:22:15.922201 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:15.922165 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66fc9bc5dc-c7k2t_bb85e53b-69b8-4562-8e76-d66b23610895/kube-auth-proxy/0.log" Sep 16 14:22:16.397085 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:16.397059 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4kn2r_485872f2-06f0-40e9-99a9-f603cd536dd4/payload-processing/0.log" Sep 16 14:22:16.508063 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:16.508032 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcdcbb9b-kbf6r_5520568f-3a4f-4665-b75c-d4726ccfebb1/router/0.log" Sep 16 14:22:16.856227 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:16.856193 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-nhptf_41562f83-cf4e-4b0d-8a0f-250683374aa7/main/0.log" Sep 16 14:22:17.098558 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:17.098521 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-68p7b_5389af03-dc57-4ecc-a4ef-bc3fbb85ebbf/main/0.log" Sep 16 14:22:17.219764 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:17.219717 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-fw2sd_1012cac1-5ca3-4865-8b18-dec6624176e4/main/0.log" Sep 16 14:22:26.849748 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:26.849715 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hddml_1499dfa1-b0ce-44a0-add8-c1404440ecdd/global-pull-secret-syncer/0.log" Sep 16 14:22:27.163167 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:27.163061 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zmd5t_1543153d-e6e7-44f0-881e-9f78070b70c7/konnectivity-agent/0.log" Sep 16 14:22:27.257236 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:27.257201 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-222.ec2.internal_039e38a7f72228dfa18bca5c2b7c263f/haproxy/0.log" Sep 16 14:22:32.139529 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:32.139497 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-nvld6_b83b14bf-169c-4d36-8a6a-e3098c9c0841/manager/0.log" Sep 16 14:22:33.929408 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:33.929378 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-8gtcr_e2f131fb-a923-428d-82eb-39376dbd5594/cluster-monitoring-operator/0.log" Sep 16 14:22:34.066427 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:34.066388 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76f655d865-c75lb_8686d26c-77d7-4f8e-93dd-ca5d330dd679/metrics-server/0.log" Sep 16 14:22:34.110523 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:34.110496 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-bb8mw_9036dbc1-fab1-4f82-884d-5f012481128a/monitoring-plugin/0.log" Sep 16 14:22:34.276730 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:34.276700 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q6bnh_2faeae41-30b0-474c-b2b9-89109774172e/node-exporter/0.log" Sep 16 14:22:34.304221 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:34.304187 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q6bnh_2faeae41-30b0-474c-b2b9-89109774172e/kube-rbac-proxy/0.log" Sep 16 14:22:34.338530 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:34.338510 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q6bnh_2faeae41-30b0-474c-b2b9-89109774172e/init-textfile/0.log" Sep 16 14:22:35.169303 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:35.169273 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-hxdlt_e939fe78-e2d3-4469-9b2f-4a9f1b769ccf/prometheus-operator-admission-webhook/0.log" Sep 16 14:22:35.225833 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:35.225807 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55fc589b5d-8m9rl_42f70be2-7a7a-4cd8-b95d-bfec9420a78d/telemeter-client/0.log" Sep 16 14:22:35.258195 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:35.258172 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55fc589b5d-8m9rl_42f70be2-7a7a-4cd8-b95d-bfec9420a78d/reload/0.log" Sep 16 14:22:35.290257 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:35.290237 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55fc589b5d-8m9rl_42f70be2-7a7a-4cd8-b95d-bfec9420a78d/kube-rbac-proxy/0.log" Sep 16 14:22:37.534599 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.534564 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj"] Sep 16 14:22:37.534967 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.534949 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e0461a8-b839-4c98-acf6-baed82905c86" containerName="maas-api" Sep 16 14:22:37.534967 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.534960 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e0461a8-b839-4c98-acf6-baed82905c86" containerName="maas-api" Sep 16 14:22:37.535039 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.535024 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e0461a8-b839-4c98-acf6-baed82905c86" containerName="maas-api" Sep 16 14:22:37.538243 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.538226 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.543379 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.543355 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9xhpr\"/\"default-dockercfg-wm7b4\"" Sep 16 14:22:37.544774 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.544509 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xhpr\"/\"openshift-service-ca.crt\"" Sep 16 14:22:37.545567 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.545548 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xhpr\"/\"kube-root-ca.crt\"" Sep 16 14:22:37.557441 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.557417 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj"] Sep 16 14:22:37.690506 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.690481 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:22:37.698109 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.698087 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/2.log" Sep 16 14:22:37.707107 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.707082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-sys\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.707222 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.707167 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-lib-modules\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.707277 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.707234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l45zc\" (UniqueName: \"kubernetes.io/projected/774d9282-1814-408d-9076-6f9f2ecff07d-kube-api-access-l45zc\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.707277 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.707260 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-proc\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.707383 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.707290 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-podres\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.808632 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.808568 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l45zc\" (UniqueName: \"kubernetes.io/projected/774d9282-1814-408d-9076-6f9f2ecff07d-kube-api-access-l45zc\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.808632 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.808600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-proc\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.808791 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.808632 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-podres\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.808791 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.808676 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-sys\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.808791 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.808691 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-proc\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.808791 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.808747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-lib-modules\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.808791 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.808765 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-sys\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.808983 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.808849 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-lib-modules\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.808983 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.808846 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/774d9282-1814-408d-9076-6f9f2ecff07d-podres\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.818559 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.818530 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l45zc\" (UniqueName: \"kubernetes.io/projected/774d9282-1814-408d-9076-6f9f2ecff07d-kube-api-access-l45zc\") pod \"perf-node-gather-daemonset-q2fkj\" (UID: \"774d9282-1814-408d-9076-6f9f2ecff07d\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:37.848506 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:37.848486 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:38.126571 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:38.126546 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj"] Sep 16 14:22:38.127980 ip-10-0-143-222 kubenswrapper[2581]: W0916 14:22:38.127947 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod774d9282_1814_408d_9076_6f9f2ecff07d.slice/crio-eb7044511c1d2cf8694ce34dcab72c5713e67f1c9588368e42472b34e75f084e WatchSource:0}: Error finding container eb7044511c1d2cf8694ce34dcab72c5713e67f1c9588368e42472b34e75f084e: Status 404 returned error can't find the container with id eb7044511c1d2cf8694ce34dcab72c5713e67f1c9588368e42472b34e75f084e Sep 16 14:22:38.336211 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:38.336186 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77867fd8bc-2zfrg_1e422914-f6a8-4b84-b439-45c6e72a16df/console/0.log" Sep 16 14:22:38.400405 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:38.400323 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" event={"ID":"774d9282-1814-408d-9076-6f9f2ecff07d","Type":"ContainerStarted","Data":"6dff09d17faf18d463c84c86c72811b7969073c154f0b63b34f3ffaa8cced003"} Sep 16 14:22:38.400405 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:38.400370 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" event={"ID":"774d9282-1814-408d-9076-6f9f2ecff07d","Type":"ContainerStarted","Data":"eb7044511c1d2cf8694ce34dcab72c5713e67f1c9588368e42472b34e75f084e"} Sep 16 14:22:38.400405 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:38.400401 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:38.424730 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:38.424666 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" podStartSLOduration=1.4246476750000001 podStartE2EDuration="1.424647675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:38.419916112 +0000 UTC m=+2392.383274939" watchObservedRunningTime="2026-09-16 14:22:38.424647675 +0000 UTC m=+2392.388006503" Sep 16 14:22:39.071663 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:39.071633 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-hslct_9c5c390b-795a-4aab-b5b0-b4dad8c51074/volume-data-source-validator/0.log" Sep 16 14:22:40.577566 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:40.577533 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t2dsl_039c92e5-b92f-4090-867c-5ab2dd1ed94c/dns/0.log" Sep 16 14:22:40.631028 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:40.631007 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t2dsl_039c92e5-b92f-4090-867c-5ab2dd1ed94c/kube-rbac-proxy/0.log" Sep 16 14:22:40.677469 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:40.677449 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gvnzs_cd6779bd-b278-4b06-a7ca-80a4aae73ae0/dns-node-resolver/0.log" Sep 16 14:22:41.531007 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:41.530900 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n97z8_f4305a9e-2784-4ecd-9462-64b45a220538/node-ca/0.log" Sep 16 14:22:43.387087 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:43.387054 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vxrxf_dff9285c-d48d-437a-8a49-97faee1752d0/discovery/0.log" Sep 16 14:22:43.478207 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:43.478170 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66fc9bc5dc-c7k2t_bb85e53b-69b8-4562-8e76-d66b23610895/kube-auth-proxy/0.log" Sep 16 14:22:43.663438 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:43.663363 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4kn2r_485872f2-06f0-40e9-99a9-f603cd536dd4/payload-processing/0.log" Sep 16 14:22:43.690474 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:43.690441 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcdcbb9b-kbf6r_5520568f-3a4f-4665-b75c-d4726ccfebb1/router/0.log" Sep 16 14:22:44.416802 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:44.416777 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-q2fkj" Sep 16 14:22:44.656657 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:44.656627 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5brjf_15c5b25d-bb84-4667-b18e-30771d4c1c25/serve-healthcheck-canary/0.log" Sep 16 14:22:45.337981 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:45.337949 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-6nvrv_97f4c753-e774-452c-8c8d-4070badd940e/insights-operator/0.log" Sep 16 14:22:45.338554 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:45.338533 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-6nvrv_97f4c753-e774-452c-8c8d-4070badd940e/insights-operator/1.log" Sep 16 14:22:45.532310 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:45.532282 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w7g9k_3d7a8d13-6484-443a-805b-061bbfdb8e19/kube-rbac-proxy/0.log" Sep 16 14:22:45.559485 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:45.559460 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w7g9k_3d7a8d13-6484-443a-805b-061bbfdb8e19/exporter/0.log" Sep 16 14:22:45.588357 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:45.588286 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w7g9k_3d7a8d13-6484-443a-805b-061bbfdb8e19/extractor/0.log" Sep 16 14:22:46.762566 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:46.762542 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:22:46.772308 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:46.772285 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 14:22:46.773506 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:46.773487 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qsvk6_8736f18a-8bcd-4118-a84b-8d6526243746/console-operator/1.log" Sep 16 14:22:46.781715 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:46.781699 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g58wg_23cd4ca2-227a-4011-ba43-5afafff1d0b9/ovn-acl-logging/0.log" Sep 16 14:22:48.662826 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:48.662736 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4gx77_4d68b82b-d7a5-44dc-a602-229ecfe490c7/manager/0.log" Sep 16 14:22:48.671860 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:48.671818 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vxrxf_dff9285c-d48d-437a-8a49-97faee1752d0/discovery/0.log" Sep 16 14:22:48.685772 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:48.685746 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66fc9bc5dc-c7k2t_bb85e53b-69b8-4562-8e76-d66b23610895/kube-auth-proxy/0.log" Sep 16 14:22:48.737047 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:48.737019 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d96bd79db-gd7cr_e75432b4-0b09-48dc-9bc1-7d69fdab6030/manager/0.log" Sep 16 14:22:48.806187 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:48.806134 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4kn2r_485872f2-06f0-40e9-99a9-f603cd536dd4/payload-processing/0.log" Sep 16 14:22:48.828362 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:48.828334 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcdcbb9b-kbf6r_5520568f-3a4f-4665-b75c-d4726ccfebb1/router/0.log" Sep 16 14:22:49.103928 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:49.103893 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-wm9r4_382e3f4d-0714-4b70-8e27-d471c525de0e/manager/0.log" Sep 16 14:22:49.140537 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:49.140513 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-wm9r4_382e3f4d-0714-4b70-8e27-d471c525de0e/copy-manifests/0.log" Sep 16 14:22:50.625043 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:50.625011 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hdwlg_939c43b1-48b9-4d93-977c-21d5e91e6483/openshift-lws-operator/0.log" Sep 16 14:22:51.207773 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:51.207739 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vxrxf_dff9285c-d48d-437a-8a49-97faee1752d0/discovery/0.log" Sep 16 14:22:51.232687 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:51.232666 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66fc9bc5dc-c7k2t_bb85e53b-69b8-4562-8e76-d66b23610895/kube-auth-proxy/0.log" Sep 16 14:22:51.355723 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:51.355690 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4kn2r_485872f2-06f0-40e9-99a9-f603cd536dd4/payload-processing/0.log" Sep 16 14:22:51.372134 ip-10-0-143-222 kubenswrapper[2581]: I0916 14:22:51.372107 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcdcbb9b-kbf6r_5520568f-3a4f-4665-b75c-d4726ccfebb1/router/0.log"