Sep 17 01:37:16.133902 ip-10-0-143-174 systemd[1]: Starting Kubernetes Kubelet... Sep 17 01:37:16.566677 ip-10-0-143-174 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 01:37:16.566677 ip-10-0-143-174 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 01:37:16.566677 ip-10-0-143-174 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 01:37:16.566677 ip-10-0-143-174 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 01:37:16.566677 ip-10-0-143-174 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 01:37:16.567759 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.567676 2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 01:37:16.573731 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573711 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:16.573731 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573728 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:16.573731 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573732 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:16.573731 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573735 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:16.573731 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573739 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573742 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573744 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573747 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573750 2586 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573752 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573755 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573757 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573760 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573762 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573765 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573767 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573769 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573772 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573774 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573777 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573779 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573782 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573784 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573787 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:16.573925 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573789 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573792 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573794 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573797 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573799 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573801 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573804 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573806 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573808 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573811 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573813 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573816 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573818 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573821 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573824 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573826 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573829 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573831 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573833 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573836 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:16.574428 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573838 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573840 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573843 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573845 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573848 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573850 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573853 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573857 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573860 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573863 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573866 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573869 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573872 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573874 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573877 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573880 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573882 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573885 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573887 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:16.574945 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573889 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573892 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573894 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573897 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573899 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573903 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573906 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573909 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573912 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573915 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573918 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573920 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573923 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573926 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573929 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573931 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573934 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573937 2586 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573939 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:16.575413 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573942 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573945 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573947 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573950 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573952 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573955 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573958 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573961 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573963 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573965 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573969 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573972 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573974 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573976 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573979 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573981 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.573985 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574712 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574721 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:16.575890 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574724 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574727 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574730 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574732 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574735 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574737 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574740 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574742 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574745 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574748 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574751 2586 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574753 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574756 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574759 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574761 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574765 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574767 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574770 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574773 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:16.576427 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574777 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574781 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574784 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574787 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574789 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574792 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574795 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574797 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574799 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574802 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574805 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574807 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574810 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574812 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574815 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574818 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574820 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574823 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574825 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574827 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:16.576927 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574830 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574832 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574835 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574837 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574840 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574842 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574845 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574847 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574849 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574852 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574855 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574858 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574860 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574863 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574865 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574868 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574871 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574873 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574875 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574878 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:16.577420 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574880 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574883 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574887 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574891 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574894 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574897 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574900 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574902 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574905 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574907 2586 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574910 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574912 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574915 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574917 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574919 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574922 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574924 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574930 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574933 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:16.577904 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574936 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574938 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574941 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574944 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574946 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574949 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574952 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574955 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574957 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574960 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574962 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574965 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574967 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574970 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574972 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574975 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574977 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574980 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.574983 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575373 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:16.578386 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575377 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575380 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575384 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575387 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575390 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575393 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575396 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575399 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575401 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575406 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575409 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575412 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575415 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575418 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575421 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575425 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575428 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575431 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575434 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:16.578901 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575436 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575439 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575441 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575446 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575461 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575466 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575469 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575472 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575475 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575478 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575484 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575487 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575490 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575493 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575495 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575497 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575500 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575504 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:16.579380 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575507 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575509 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575511 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575514 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575516 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575518 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575520 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575523 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575525 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575527 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575530 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575534 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575536 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575539 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575541 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575544 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575546 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575549 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575551 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575554 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:16.579835 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575556 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575558 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575561 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575564 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575567 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575570 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575572 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575574 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575592 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575595 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575598 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575600 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575603 2586 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575605 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575610 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575612 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575614 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575617 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575619 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575622 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:16.580303 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575624 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575626 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575629 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575631 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575634 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575637 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575639 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575641 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575644 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575646 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575649 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575651 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575654 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575656 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575658 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575661 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575665 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575667 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575670 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575672 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:16.580824 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.575674 2586 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575747 2586 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575784 2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575790 2586 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575795 2586 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575803 2586 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575808 2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575812 2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575816 2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575819 2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575822 2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575825 2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575828 2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575831 2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575834 2586 flags.go:64] FLAG: --cgroup-root="" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575837 2586 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575839 2586 flags.go:64] FLAG: --client-ca-file="" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575843 2586 flags.go:64] FLAG: --cloud-provider="external" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575846 2586 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575850 2586 flags.go:64] FLAG: --cluster-domain="" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575852 2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575855 2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575858 2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575861 2586 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 01:37:16.581290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575865 2586 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575868 2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575870 2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575873 2586 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575878 2586 flags.go:64] FLAG: --contention-profiling="false" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575881 2586 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575883 2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575886 2586 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575889 2586 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575893 2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575896 2586 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575899 2586 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575902 2586 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575905 2586 flags.go:64] FLAG: --enable-server="true" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575909 2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575913 2586 flags.go:64] FLAG: --event-burst="100" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575916 2586 flags.go:64] FLAG: --event-qps="50" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575919 2586 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575922 2586 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575925 2586 flags.go:64] FLAG: --eviction-hard="" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575928 2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575932 2586 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575935 2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575938 2586 flags.go:64] FLAG: --eviction-soft="" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575940 2586 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 01:37:16.581952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575943 2586 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575946 2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575949 2586 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575952 2586 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575955 2586 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575957 2586 flags.go:64] FLAG: --feature-gates="" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575961 2586 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575964 2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575967 2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575970 2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575973 2586 flags.go:64] FLAG: --healthz-port="10248" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575977 2586 flags.go:64] FLAG: --help="false" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575980 2586 flags.go:64] FLAG: --hostname-override="ip-10-0-143-174.ec2.internal" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575982 2586 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575985 2586 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575988 2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575991 2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575994 2586 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575997 2586 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.575999 2586 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576002 2586 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576006 2586 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576009 2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576012 2586 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 01:37:16.582613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576015 2586 flags.go:64] FLAG: --kube-reserved="" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576017 2586 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576020 2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576023 2586 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576025 2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576028 2586 flags.go:64] FLAG: --lock-file="" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576031 2586 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576033 2586 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576036 2586 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576041 2586 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576058 2586 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576062 2586 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576064 2586 flags.go:64] FLAG: --logging-format="text" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576067 2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576070 2586 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576073 2586 flags.go:64] FLAG: --manifest-url="" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576075 2586 flags.go:64] FLAG: --manifest-url-header="" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576087 2586 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576089 2586 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576094 2586 flags.go:64] FLAG: --max-pods="110" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576097 2586 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576100 2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576103 2586 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576106 2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576108 2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 01:37:16.583222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576111 2586 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576114 2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576121 2586 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576123 2586 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576127 2586 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576130 2586 flags.go:64] FLAG: --pod-cidr="" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576133 2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576139 2586 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576141 2586 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576144 2586 flags.go:64] FLAG: --pods-per-core="0" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576146 2586 flags.go:64] FLAG: --port="10250" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576149 2586 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576153 2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d7b57c4afd84a820" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576156 2586 flags.go:64] FLAG: --qos-reserved="" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576158 2586 flags.go:64] FLAG: --read-only-port="10255" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576161 2586 flags.go:64] FLAG: --register-node="true" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576164 2586 flags.go:64] FLAG: --register-with-taints="" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576167 2586 flags.go:64] FLAG: --registry-burst="10" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576171 2586 flags.go:64] FLAG: --registry-qps="5" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576173 2586 flags.go:64] FLAG: --reserved-cpus="" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576176 2586 flags.go:64] FLAG: --reserved-memory="" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576180 2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576182 2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576185 2586 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576188 2586 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 01:37:16.583876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576190 2586 flags.go:64] FLAG: --runonce="false" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576193 2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576198 2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576201 2586 flags.go:64] FLAG: --seccomp-default="false" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576203 2586 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576206 2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576209 2586 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576211 2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576214 2586 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576217 2586 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576219 2586 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576224 2586 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576227 2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576230 2586 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576233 2586 flags.go:64] FLAG: --system-cgroups="" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576236 2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576241 2586 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576244 2586 flags.go:64] FLAG: --tls-cert-file="" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576246 2586 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576250 2586 flags.go:64] FLAG: --tls-min-version="" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576253 2586 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576255 2586 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576258 2586 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576261 2586 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576263 2586 flags.go:64] FLAG: --v="2" Sep 17 01:37:16.584509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576267 2586 flags.go:64] FLAG: --version="false" Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576272 2586 flags.go:64] FLAG: --vmodule="" Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576275 2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.576278 2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576393 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576396 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576399 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576402 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576405 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576409 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576411 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576413 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576416 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576418 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576421 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576423 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576426 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576430 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576435 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576439 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:16.585128 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576442 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576445 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576447 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576450 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576452 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576455 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576458 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576460 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576463 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576465 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576468 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576470 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576473 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576475 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576478 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576481 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576483 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576485 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576487 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576490 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:16.585641 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576492 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576496 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576499 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576501 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576504 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576506 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576508 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576511 2586 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576513 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576517 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576522 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576525 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576528 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576530 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576532 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576535 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576537 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576539 2586 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576542 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576544 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:16.586127 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576547 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576549 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576551 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576554 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576556 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576558 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576561 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576563 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576566 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576568 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576570 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576573 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576591 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576596 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576599 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576602 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576605 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576608 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576611 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576613 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:16.586674 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576616 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576619 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576622 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576625 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576627 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576630 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576632 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576634 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576637 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576639 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576641 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576644 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576646 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576649 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576651 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576653 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576656 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576658 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576661 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:16.587176 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576663 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576666 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576669 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.576671 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.577389 2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.585356 2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.585486 2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.585501 2586 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.585506 2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585560 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585565 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585568 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585571 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585574 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585591 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:16.587770 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585594 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585597 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585600 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585602 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585605 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585607 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585610 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585612 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585615 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585618 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585620 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585623 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585625 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585628 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585630 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585633 2586 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585635 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585638 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585640 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585643 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:16.588141 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585645 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585648 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585651 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585654 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585657 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585660 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585662 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585665 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585667 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585670 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585672 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585675 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585677 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585680 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585682 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585684 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585687 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585690 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585692 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:16.588631 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585694 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585697 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585700 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585703 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585706 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585708 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585711 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585714 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585716 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585719 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585721 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585723 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585726 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585730 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585733 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585736 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585739 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585742 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585745 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:16.589120 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585748 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585752 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585756 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585758 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585761 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585764 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585766 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585768 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585771 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585773 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585776 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585780 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585783 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585786 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585789 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585792 2586 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585794 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585797 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585799 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:16.589569 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585802 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585805 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585807 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585810 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585812 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585815 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585817 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585820 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585822 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585825 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585828 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585830 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585833 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585836 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585839 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:16.590033 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585841 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.585846 2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585954 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585959 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585962 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585964 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585967 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585970 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585972 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585975 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585977 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585980 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585982 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585985 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585988 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:16.590417 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585991 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585993 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.585996 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586000 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586003 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586006 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586009 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586012 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586014 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586017 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586020 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586022 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586025 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586027 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586030 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586033 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586036 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586038 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586040 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:16.590819 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586043 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586045 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586048 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586051 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586054 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586056 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586059 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586061 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586064 2586 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586066 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586068 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586071 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586073 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586077 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586080 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586083 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586085 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586088 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586091 2586 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:16.591280 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586093 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586096 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586098 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586100 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586103 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586105 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586107 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586110 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586112 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586115 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586118 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586120 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586123 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586125 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586127 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586130 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586132 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586135 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586138 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586140 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:16.591815 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586143 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586145 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586148 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586151 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586153 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586155 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586158 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586160 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586163 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586165 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586168 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586170 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586173 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586175 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586178 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586180 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586183 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586185 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586188 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586191 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:16.592285 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586193 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586196 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586198 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586201 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586203 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586206 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586208 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:16.586211 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.586215 2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.586323 2586 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.588689 2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.589806 2586 server.go:1019] "Starting client certificate rotation" Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.589899 2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 01:37:16.592762 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.589936 2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 01:37:16.593289 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.593276 2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 01:37:16.597720 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.597699 2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 01:37:16.607645 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.607628 2586 log.go:25] "Validated CRI v1 runtime API" Sep 17 01:37:16.611639 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.611624 2586 log.go:25] "Validated CRI v1 image API" Sep 17 01:37:16.612536 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.612518 2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 01:37:16.617512 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.617492 2586 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 dba99888-4102-4d41-81a4-9fb8baf0c081:/dev/nvme0n1p4 ff5b2cca-3e04-4c43-b4dc-0a087bc7de1d:/dev/nvme0n1p3] Sep 17 01:37:16.617561 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.617511 2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 01:37:16.620757 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.620740 2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 01:37:16.621792 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.621685 2586 manager.go:216] Machine: {Timestamp:2026-09-17 01:37:16.621182961 +0000 UTC m=+0.377950202 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3197001 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ec0a315ac39ab5304a0fad20a7c11 SystemUUID:ec2ec0a3-15ac-39ab-5304-a0fad20a7c11 BootID:f07afb83-7cc4-4a17-b41c-1c9747a98b62 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fd:5d:2b:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fd:5d:2b:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:fc:8f:69:de:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 01:37:16.621792 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.621789 2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 01:37:16.621908 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.621897 2586 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 01:37:16.624871 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.624849 2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 01:37:16.625006 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.624875 2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-174.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 01:37:16.626421 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.626410 2586 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 01:37:16.626459 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.626423 2586 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 01:37:16.626459 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.626430 2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 01:37:16.627513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.627503 2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 01:37:16.628699 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.628690 2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 01:37:16.633324 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.633308 2586 state_mem.go:36] "Initialized new in-memory state store" Sep 17 01:37:16.633608 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.633596 2586 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 01:37:16.633672 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.633630 2586 kubelet.go:486] "Attempting to sync node with API server" Sep 17 01:37:16.633672 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.633644 2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 01:37:16.633672 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.633655 2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 01:37:16.633672 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.633665 2586 kubelet.go:398] "Adding apiserver pod source" Sep 17 01:37:16.633672 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.633673 2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 01:37:16.634154 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.634006 2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 01:37:16.634154 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.634023 2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 01:37:16.636817 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.636795 2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 17 01:37:16.639791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.639778 2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 01:37:16.639849 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.639807 2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 01:37:16.642644 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642630 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 01:37:16.642702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642651 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 01:37:16.642702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642658 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 01:37:16.642702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642663 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 01:37:16.642702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642673 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 01:37:16.642702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642679 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 01:37:16.642702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642692 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 01:37:16.642702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642697 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 01:37:16.642702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642705 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 01:37:16.642909 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642710 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 01:37:16.642909 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642719 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 01:37:16.642909 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.642725 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 01:37:16.644278 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.644266 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 01:37:16.644278 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.644278 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 01:37:16.645153 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.645131 2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-174.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 01:37:16.645210 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.645130 2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 01:37:16.648319 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.648299 2586 server.go:1268] "Started kubelet" Sep 17 01:37:16.649207 ip-10-0-143-174 systemd[1]: Started Kubernetes Kubelet. Sep 17 01:37:16.649482 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.649218 2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 01:37:16.649482 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.649407 2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 01:37:16.649602 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.649548 2586 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 01:37:16.651023 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.651002 2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 01:37:16.651481 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.651465 2586 server.go:310] "Adding debug handlers to kubelet server" Sep 17 01:37:16.656888 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.656862 2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 01:37:16.658433 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.658417 2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 01:37:16.659129 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.659112 2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 01:37:16.659354 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.659334 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-174.ec2.internal\" not found" Sep 17 01:37:16.659886 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.659862 2586 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 01:37:16.659886 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.659881 2586 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 01:37:16.659998 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.659859 2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 01:37:16.659998 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.659960 2586 reconstruct.go:97] "Volume reconstruction finished" Sep 17 01:37:16.659998 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.659968 2586 reconciler.go:29] "Reconciler: start to sync state" Sep 17 01:37:16.660278 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.660263 2586 factory.go:153] Registering CRI-O factory Sep 17 01:37:16.660356 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.660303 2586 factory.go:223] Registration of the crio container factory successfully Sep 17 01:37:16.660405 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.660373 2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 01:37:16.660405 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.660384 2586 factory.go:55] Registering systemd factory Sep 17 01:37:16.660405 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.660393 2586 factory.go:223] Registration of the systemd container factory successfully Sep 17 01:37:16.660528 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.660410 2586 factory.go:103] Registering Raw factory Sep 17 01:37:16.660528 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.660426 2586 manager.go:1196] Started watching for new ooms in manager Sep 17 01:37:16.660919 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.660903 2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-174.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 01:37:16.661479 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.661456 2586 manager.go:349] Starting recovery of all containers Sep 17 01:37:16.662136 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.660761 2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-174.ec2.internal.18d5f7efe43f9993 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-174.ec2.internal,UID:ip-10-0-143-174.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-174.ec2.internal,},FirstTimestamp:2026-09-17 01:37:16.648257939 +0000 UTC m=+0.405025176,LastTimestamp:2026-09-17 01:37:16.648257939 +0000 UTC m=+0.405025176,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-174.ec2.internal,}" Sep 17 01:37:16.666367 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.666324 2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-174.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 01:37:16.666716 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.666658 2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 01:37:16.669717 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.669681 2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 01:37:16.671709 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.671692 2586 manager.go:354] Recovery completed Sep 17 01:37:16.676007 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.675995 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:16.677682 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.677653 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:16.677729 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.677697 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:16.677729 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.677711 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:16.677995 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.677979 2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hswrx" Sep 17 01:37:16.678240 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.678228 2586 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 01:37:16.678297 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.678240 2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 01:37:16.678297 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.678247 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:16.678297 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.678257 2586 state_mem.go:36] "Initialized new in-memory state store" Sep 17 01:37:16.678965 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.678950 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:16.679027 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.678982 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:16.679027 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.678994 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:16.680052 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.679988 2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-174.ec2.internal.18d5f7efe600923c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-174.ec2.internal,UID:ip-10-0-143-174.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-174.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-174.ec2.internal,},FirstTimestamp:2026-09-17 01:37:16.677681724 +0000 UTC m=+0.434448958,LastTimestamp:2026-09-17 01:37:16.677681724 +0000 UTC m=+0.434448958,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-174.ec2.internal,}" Sep 17 01:37:16.680471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.680459 2586 policy_none.go:49] "None policy: Start" Sep 17 01:37:16.680530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.680478 2586 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 01:37:16.680530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.680491 2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 01:37:16.682553 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.682539 2586 policy_none.go:47] "Start" Sep 17 01:37:16.684606 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.684565 2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hswrx" Sep 17 01:37:16.689494 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.689429 2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-174.ec2.internal.18d5f7efe600ebff default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-174.ec2.internal,UID:ip-10-0-143-174.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-174.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-174.ec2.internal,},FirstTimestamp:2026-09-17 01:37:16.677704703 +0000 UTC m=+0.434471937,LastTimestamp:2026-09-17 01:37:16.677704703 +0000 UTC m=+0.434471937,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-174.ec2.internal,}" Sep 17 01:37:16.716738 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.716727 2586 manager.go:337] "Starting Device Plugin manager" Sep 17 01:37:16.742603 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.716770 2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 01:37:16.742603 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.716783 2586 server.go:87] "Starting device plugin registration server" Sep 17 01:37:16.742603 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.717099 2586 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 01:37:16.742603 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.717110 2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 01:37:16.742603 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.717231 2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 01:37:16.742603 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.717305 2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 01:37:16.742603 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.717314 2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 01:37:16.742603 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.718157 2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 01:37:16.742603 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.718202 2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-174.ec2.internal\" not found" Sep 17 01:37:16.760784 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.760758 2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 01:37:16.760887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.760817 2586 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 01:37:16.760887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.760839 2586 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 01:37:16.760887 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.760873 2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 01:37:16.762992 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.762973 2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:16.818221 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.818113 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:16.819183 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.819159 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:16.819297 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.819192 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:16.819297 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.819203 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:16.819297 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.819227 2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-174.ec2.internal" Sep 17 01:37:16.828209 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.828192 2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-174.ec2.internal" Sep 17 01:37:16.828298 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.828215 2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-174.ec2.internal\": node \"ip-10-0-143-174.ec2.internal\" not found" Sep 17 01:37:16.857664 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.857642 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-174.ec2.internal\" not found" Sep 17 01:37:16.861876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.861859 2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal"] Sep 17 01:37:16.861955 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.861931 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:16.863330 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.863314 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:16.863432 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.863346 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:16.863432 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.863391 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:16.865714 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.865700 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:16.865870 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.865856 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" Sep 17 01:37:16.865927 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.865884 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:16.866418 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.866401 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:16.866506 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.866447 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:16.866506 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.866457 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:16.866506 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.866481 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:16.866506 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.866494 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:16.866506 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.866461 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:16.868764 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.868747 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal" Sep 17 01:37:16.868855 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.868770 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:16.869439 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.869425 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:16.869515 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.869449 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:16.869515 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.869460 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:16.891287 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.891266 2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-174.ec2.internal\" not found" node="ip-10-0-143-174.ec2.internal" Sep 17 01:37:16.895755 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.895739 2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-174.ec2.internal\" not found" node="ip-10-0-143-174.ec2.internal" Sep 17 01:37:16.958301 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:16.958281 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-174.ec2.internal\" not found" Sep 17 01:37:16.961617 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.961598 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/65ada5008fdb1ae2d02491334493ce88-config\") pod \"kube-apiserver-proxy-ip-10-0-143-174.ec2.internal\" (UID: \"65ada5008fdb1ae2d02491334493ce88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal" Sep 17 01:37:16.961694 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.961632 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5959b59db0da16a9835d45cb187dd20d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal\" (UID: \"5959b59db0da16a9835d45cb187dd20d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" Sep 17 01:37:16.961694 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:16.961657 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5959b59db0da16a9835d45cb187dd20d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal\" (UID: \"5959b59db0da16a9835d45cb187dd20d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.059091 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.059045 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-174.ec2.internal\" not found" Sep 17 01:37:17.062341 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.062320 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5959b59db0da16a9835d45cb187dd20d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal\" (UID: \"5959b59db0da16a9835d45cb187dd20d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.062454 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.062373 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5959b59db0da16a9835d45cb187dd20d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal\" (UID: \"5959b59db0da16a9835d45cb187dd20d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.062454 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.062400 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/65ada5008fdb1ae2d02491334493ce88-config\") pod \"kube-apiserver-proxy-ip-10-0-143-174.ec2.internal\" (UID: \"65ada5008fdb1ae2d02491334493ce88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.062454 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.062425 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5959b59db0da16a9835d45cb187dd20d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal\" (UID: \"5959b59db0da16a9835d45cb187dd20d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.062565 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.062454 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/65ada5008fdb1ae2d02491334493ce88-config\") pod \"kube-apiserver-proxy-ip-10-0-143-174.ec2.internal\" (UID: \"65ada5008fdb1ae2d02491334493ce88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.062565 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.062493 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5959b59db0da16a9835d45cb187dd20d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal\" (UID: \"5959b59db0da16a9835d45cb187dd20d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.159863 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.159792 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-174.ec2.internal\" not found" Sep 17 01:37:17.194216 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.194186 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.199651 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.199249 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.260223 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.260190 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-174.ec2.internal\" not found" Sep 17 01:37:17.360722 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.360692 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-174.ec2.internal\" not found" Sep 17 01:37:17.460043 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.459961 2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:17.482162 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.482134 2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:17.560026 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.559980 2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.572218 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.572195 2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 01:37:17.573120 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.573103 2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal" Sep 17 01:37:17.580853 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.580837 2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 01:37:17.590036 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.590019 2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 01:37:17.590153 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.590136 2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 01:37:17.590206 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.590173 2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 01:37:17.634689 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.634662 2586 apiserver.go:52] "Watching apiserver" Sep 17 01:37:17.642217 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.642195 2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 01:37:17.642549 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.642528 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-bjxzr","openshift-network-diagnostics/network-check-target-rs9g5","openshift-network-operator/iptables-alerter-crpps","kube-system/konnectivity-agent-wvqr2","kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal","openshift-dns/node-resolver-k5mbc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal","openshift-multus/multus-additional-cni-plugins-kxzh6","openshift-ovn-kubernetes/ovnkube-node-rw7gk","kube-system/global-pull-secret-syncer-f62th","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq","openshift-cluster-node-tuning-operator/tuned-9lpk2","openshift-image-registry/node-ca-rf6rr","openshift-multus/multus-xlplg"] Sep 17 01:37:17.644997 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.644984 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:17.645063 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.645041 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:17.647128 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.647110 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.649249 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.649232 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:17.649414 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.649396 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 01:37:17.649596 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.649556 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 01:37:17.649678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.649640 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 01:37:17.649678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.649643 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zwql2\"" Sep 17 01:37:17.651344 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.651322 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:37:17.651392 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.651368 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:17.651548 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.651533 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-254nl\"" Sep 17 01:37:17.652402 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.652388 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 01:37:17.652467 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.652407 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 01:37:17.653553 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.653539 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 01:37:17.653667 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.653638 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.653945 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.653927 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 01:37:17.654042 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.654016 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b9vck\"" Sep 17 01:37:17.655673 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.655654 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 01:37:17.655843 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.655832 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2bsqq\"" Sep 17 01:37:17.655911 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.655897 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:17.656200 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.656009 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 01:37:17.656200 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.656037 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 01:37:17.656200 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.656135 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 01:37:17.657869 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.657854 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jsb6n\"" Sep 17 01:37:17.657925 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.657885 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 01:37:17.657969 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.657885 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 01:37:17.658232 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.658219 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.658501 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.658481 2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 01:37:17.660521 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.660499 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.661372 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.661354 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 01:37:17.661372 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.661367 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 01:37:17.661512 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.661358 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 01:37:17.661512 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.661399 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 01:37:17.661512 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.661428 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w6c5b\"" Sep 17 01:37:17.661512 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.661376 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 01:37:17.661512 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.661371 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 01:37:17.662495 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.662480 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 01:37:17.662559 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.662497 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.662787 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.663074 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.663173 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.663485 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mc6jn\"" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.664822 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c40e778-6932-421e-871c-7654d4d5912f-cni-binary-copy\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.664857 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-multus-socket-dir-parent\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.664898 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6qng\" (UniqueName: \"kubernetes.io/projected/f8110c49-0824-494d-9f27-e0ba6e54a723-kube-api-access-x6qng\") pod \"iptables-alerter-crpps\" (UID: \"f8110c49-0824-494d-9f27-e0ba6e54a723\") " pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.664921 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-cni-netd\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.664939 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-system-cni-dir\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.664956 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-multus-conf-dir\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.664988 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c40e778-6932-421e-871c-7654d4d5912f-multus-daemon-config\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.665047 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-slash\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.665170 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-ovnkube-script-lib\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.665216 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.665244 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-var-lib-openvswitch\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.665283 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9b4c988-c58d-4e30-94e9-a892d64598b4-host\") pod \"node-ca-rf6rr\" (UID: \"d9b4c988-c58d-4e30-94e9-a892d64598b4\") " pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.665389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.665305 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-socket-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.666047 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.665381 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-run-systemd\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.666047 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.665903 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-run-ovn\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.666047 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.665938 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31ad03fc-51f4-46b5-9a17-4d7f38513e84-dbus\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:17.666047 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666017 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6x42\" (UniqueName: \"kubernetes.io/projected/d9b4c988-c58d-4e30-94e9-a892d64598b4-kube-api-access-v6x42\") pod \"node-ca-rf6rr\" (UID: \"d9b4c988-c58d-4e30-94e9-a892d64598b4\") " pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.666208 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666105 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-ovn-node-metrics-cert\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.666208 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666183 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9b4c988-c58d-4e30-94e9-a892d64598b4-serviceca\") pod \"node-ca-rf6rr\" (UID: \"d9b4c988-c58d-4e30-94e9-a892d64598b4\") " pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.666291 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666213 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrjlf\" (UniqueName: \"kubernetes.io/projected/9c40e778-6932-421e-871c-7654d4d5912f-kube-api-access-hrjlf\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.666291 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666238 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31ad03fc-51f4-46b5-9a17-4d7f38513e84-kubelet-config\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:17.666291 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666277 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:17.666452 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666326 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.666506 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666451 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-multus-cni-dir\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.666554 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666507 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8110c49-0824-494d-9f27-e0ba6e54a723-iptables-alerter-script\") pod \"iptables-alerter-crpps\" (UID: \"f8110c49-0824-494d-9f27-e0ba6e54a723\") " pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:17.666628 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666567 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-kubelet\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.666628 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.666601 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:17.666723 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666654 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-run-openvswitch\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.666723 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666684 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e7731ab-68de-44b8-8897-daeee9096d24-konnectivity-ca\") pod \"konnectivity-agent-wvqr2\" (UID: \"6e7731ab-68de-44b8-8897-daeee9096d24\") " pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:17.666723 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.666716 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-cnibin\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.667055 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667032 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-etc-kubernetes\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.667142 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667118 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrk9r\" (UniqueName: \"kubernetes.io/projected/04add3b9-e936-4e41-9562-21031e041cc4-kube-api-access-nrk9r\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.667236 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667160 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-var-lib-kubelet\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.667236 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667183 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjmw\" (UniqueName: \"kubernetes.io/projected/4c2f9cad-e927-484e-9599-a9a84ade862e-kube-api-access-vhjmw\") pod \"node-resolver-k5mbc\" (UID: \"4c2f9cad-e927-484e-9599-a9a84ade862e\") " pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:17.667236 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667206 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-systemd-units\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.667236 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667228 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-run-ovn-kubernetes\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.667385 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667254 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-cni-bin\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.667385 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667292 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6f8\" (UniqueName: \"kubernetes.io/projected/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-kube-api-access-bm6f8\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.667385 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667337 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c2f9cad-e927-484e-9599-a9a84ade862e-hosts-file\") pod \"node-resolver-k5mbc\" (UID: \"4c2f9cad-e927-484e-9599-a9a84ade862e\") " pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:17.667485 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667381 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-run-multus-certs\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.667485 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667416 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-registration-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.667485 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667438 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-run-k8s-cni-cncf-io\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.667485 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667465 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-run-netns\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.667642 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667483 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-var-lib-cni-multus\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.667642 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667502 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-hostroot\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.667642 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667516 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8110c49-0824-494d-9f27-e0ba6e54a723-host-slash\") pod \"iptables-alerter-crpps\" (UID: \"f8110c49-0824-494d-9f27-e0ba6e54a723\") " pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:17.667642 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667530 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-etc-openvswitch\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.667642 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667543 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-var-lib-cni-bin\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.667642 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667557 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-sys-fs\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.667642 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667569 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2f9cad-e927-484e-9599-a9a84ade862e-tmp-dir\") pod \"node-resolver-k5mbc\" (UID: \"4c2f9cad-e927-484e-9599-a9a84ade862e\") " pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:17.667642 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667620 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-ovnkube-config\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.667889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667650 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e7731ab-68de-44b8-8897-daeee9096d24-agent-certs\") pod \"konnectivity-agent-wvqr2\" (UID: \"6e7731ab-68de-44b8-8897-daeee9096d24\") " pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:17.667889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667679 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-device-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.667889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667697 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-run-netns\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.667889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667710 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-node-log\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.667889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667730 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-log-socket\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.667889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667752 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.667889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667767 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:17.667889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.667780 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-env-overrides\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.669335 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.669320 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.670454 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.670282 2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 01:37:17.671477 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.671447 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 01:37:17.671559 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.671517 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mdvvn\"" Sep 17 01:37:17.671761 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.671523 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 01:37:17.671800 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.671761 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.673715 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.673696 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:37:17.674028 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.674012 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlcpq\"" Sep 17 01:37:17.674233 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.674221 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 01:37:17.690134 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.690085 2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 01:32:17 +0000 UTC" deadline="2028-03-27 08:19:38.938526027 +0000 UTC" Sep 17 01:37:17.690134 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.690131 2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13374h42m21.248399143s" Sep 17 01:37:17.697008 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.696989 2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lck6v" Sep 17 01:37:17.706486 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.706468 2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lck6v" Sep 17 01:37:17.760988 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.760967 2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 01:37:17.768217 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768190 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c40e778-6932-421e-871c-7654d4d5912f-cni-binary-copy\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.768217 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768214 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-multus-socket-dir-parent\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.768374 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768231 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6qng\" (UniqueName: \"kubernetes.io/projected/f8110c49-0824-494d-9f27-e0ba6e54a723-kube-api-access-x6qng\") pod \"iptables-alerter-crpps\" (UID: \"f8110c49-0824-494d-9f27-e0ba6e54a723\") " pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:17.768374 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768253 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-cni-netd\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.768374 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768297 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-system-cni-dir\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.768374 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768316 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-multus-socket-dir-parent\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.768374 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768352 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-sys\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.768374 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768373 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-tuned\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.768658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768403 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-cni-netd\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.768658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768502 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-system-cni-dir\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.768658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768526 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-multus-conf-dir\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.768658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768545 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c40e778-6932-421e-871c-7654d4d5912f-multus-daemon-config\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.768658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768563 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-slash\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.768658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768599 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-ovnkube-script-lib\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.768658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768622 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-multus-conf-dir\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.768658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768636 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.768658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768637 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-system-cni-dir\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.769135 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768680 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-slash\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.769135 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768801 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-sysconfig\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.769135 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768828 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c40e778-6932-421e-871c-7654d4d5912f-cni-binary-copy\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.769135 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768837 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.769135 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768879 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-var-lib-openvswitch\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.769135 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768900 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.769135 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768930 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-modprobe-d\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.769135 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768978 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-var-lib-openvswitch\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.769135 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769121 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9b4c988-c58d-4e30-94e9-a892d64598b4-host\") pod \"node-ca-rf6rr\" (UID: \"d9b4c988-c58d-4e30-94e9-a892d64598b4\") " pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769147 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c40e778-6932-421e-871c-7654d4d5912f-multus-daemon-config\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769166 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-ovnkube-script-lib\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.768952 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9b4c988-c58d-4e30-94e9-a892d64598b4-host\") pod \"node-ca-rf6rr\" (UID: \"d9b4c988-c58d-4e30-94e9-a892d64598b4\") " pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769367 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-socket-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769381 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-run-systemd\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769394 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-run-ovn\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769406 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-run\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769425 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31ad03fc-51f4-46b5-9a17-4d7f38513e84-dbus\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769461 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6x42\" (UniqueName: \"kubernetes.io/projected/d9b4c988-c58d-4e30-94e9-a892d64598b4-kube-api-access-v6x42\") pod \"node-ca-rf6rr\" (UID: \"d9b4c988-c58d-4e30-94e9-a892d64598b4\") " pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769463 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-run-systemd\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769496 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-run-ovn\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769503 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-ovn-node-metrics-cert\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769527 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-sysctl-d\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.769516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769533 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31ad03fc-51f4-46b5-9a17-4d7f38513e84-dbus\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769548 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-socket-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769574 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9b4c988-c58d-4e30-94e9-a892d64598b4-serviceca\") pod \"node-ca-rf6rr\" (UID: \"d9b4c988-c58d-4e30-94e9-a892d64598b4\") " pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769615 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrjlf\" (UniqueName: \"kubernetes.io/projected/9c40e778-6932-421e-871c-7654d4d5912f-kube-api-access-hrjlf\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769694 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31ad03fc-51f4-46b5-9a17-4d7f38513e84-kubelet-config\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769724 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769750 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769777 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-multus-cni-dir\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769792 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31ad03fc-51f4-46b5-9a17-4d7f38513e84-kubelet-config\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769802 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8110c49-0824-494d-9f27-e0ba6e54a723-iptables-alerter-script\") pod \"iptables-alerter-crpps\" (UID: \"f8110c49-0824-494d-9f27-e0ba6e54a723\") " pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769825 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-kubelet\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769845 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-run-openvswitch\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769855 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-kubelet\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769826 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-multus-cni-dir\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769866 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcm4f\" (UniqueName: \"kubernetes.io/projected/26801bb3-4b53-4677-a222-d71afd225350-kube-api-access-bcm4f\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769895 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-run-openvswitch\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769896 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.770202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769920 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769952 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kss5z\" (UniqueName: \"kubernetes.io/projected/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-kube-api-access-kss5z\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.769980 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e7731ab-68de-44b8-8897-daeee9096d24-konnectivity-ca\") pod \"konnectivity-agent-wvqr2\" (UID: \"6e7731ab-68de-44b8-8897-daeee9096d24\") " pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770004 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-cnibin\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770026 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-etc-kubernetes\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770033 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9b4c988-c58d-4e30-94e9-a892d64598b4-serviceca\") pod \"node-ca-rf6rr\" (UID: \"d9b4c988-c58d-4e30-94e9-a892d64598b4\") " pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770049 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrk9r\" (UniqueName: \"kubernetes.io/projected/04add3b9-e936-4e41-9562-21031e041cc4-kube-api-access-nrk9r\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770100 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-cnibin\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770131 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp4vj\" (UniqueName: \"kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj\") pod \"network-check-target-rs9g5\" (UID: \"beb5923e-36ae-4f98-8dca-590530716c12\") " pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770154 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-sysctl-conf\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770173 2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770473 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-host\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770522 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-var-lib-kubelet\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770556 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjmw\" (UniqueName: \"kubernetes.io/projected/4c2f9cad-e927-484e-9599-a9a84ade862e-kube-api-access-vhjmw\") pod \"node-resolver-k5mbc\" (UID: \"4c2f9cad-e927-484e-9599-a9a84ade862e\") " pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770607 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-systemd-units\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770641 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-run-ovn-kubernetes\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770664 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-cni-bin\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770710 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6f8\" (UniqueName: \"kubernetes.io/projected/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-kube-api-access-bm6f8\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.771092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770717 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-etc-kubernetes\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770741 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-lib-modules\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770772 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c2f9cad-e927-484e-9599-a9a84ade862e-hosts-file\") pod \"node-resolver-k5mbc\" (UID: \"4c2f9cad-e927-484e-9599-a9a84ade862e\") " pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770801 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-run-multus-certs\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770826 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e7731ab-68de-44b8-8897-daeee9096d24-konnectivity-ca\") pod \"konnectivity-agent-wvqr2\" (UID: \"6e7731ab-68de-44b8-8897-daeee9096d24\") " pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770834 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770860 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770891 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dj6w\" (UniqueName: \"kubernetes.io/projected/05a6da2f-baeb-425b-af20-9a71e6142b87-kube-api-access-5dj6w\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770922 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-run-ovn-kubernetes\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770924 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-registration-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770971 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-systemd\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.770984 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-registration-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771006 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-run-k8s-cni-cncf-io\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771025 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-systemd-units\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771070 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-run-k8s-cni-cncf-io\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771140 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-var-lib-kubelet\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771191 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-cni-bin\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.771959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771202 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c2f9cad-e927-484e-9599-a9a84ade862e-hosts-file\") pod \"node-resolver-k5mbc\" (UID: \"4c2f9cad-e927-484e-9599-a9a84ade862e\") " pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771263 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-run-multus-certs\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771371 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-run-netns\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771407 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-var-lib-cni-multus\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771411 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8110c49-0824-494d-9f27-e0ba6e54a723-iptables-alerter-script\") pod \"iptables-alerter-crpps\" (UID: \"f8110c49-0824-494d-9f27-e0ba6e54a723\") " pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771433 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-hostroot\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771494 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-var-lib-cni-multus\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771490 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-run-netns\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771526 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8110c49-0824-494d-9f27-e0ba6e54a723-host-slash\") pod \"iptables-alerter-crpps\" (UID: \"f8110c49-0824-494d-9f27-e0ba6e54a723\") " pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771552 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-etc-openvswitch\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771554 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-hostroot\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771594 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-var-lib-kubelet\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771620 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-var-lib-cni-bin\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771648 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-sys-fs\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771674 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2f9cad-e927-484e-9599-a9a84ade862e-tmp-dir\") pod \"node-resolver-k5mbc\" (UID: \"4c2f9cad-e927-484e-9599-a9a84ade862e\") " pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771691 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c40e778-6932-421e-871c-7654d4d5912f-host-var-lib-cni-bin\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771701 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-ovnkube-config\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771755 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-etc-openvswitch\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.772790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771757 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-kubernetes\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771619 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8110c49-0824-494d-9f27-e0ba6e54a723-host-slash\") pod \"iptables-alerter-crpps\" (UID: \"f8110c49-0824-494d-9f27-e0ba6e54a723\") " pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.771807 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e7731ab-68de-44b8-8897-daeee9096d24-agent-certs\") pod \"konnectivity-agent-wvqr2\" (UID: \"6e7731ab-68de-44b8-8897-daeee9096d24\") " pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772224 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2f9cad-e927-484e-9599-a9a84ade862e-tmp-dir\") pod \"node-resolver-k5mbc\" (UID: \"4c2f9cad-e927-484e-9599-a9a84ade862e\") " pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772238 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-ovnkube-config\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772281 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-sys-fs\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772290 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-device-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772438 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04add3b9-e936-4e41-9562-21031e041cc4-device-dir\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772437 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-run-netns\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772521 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-node-log\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772557 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-run-netns\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772572 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-node-log\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772613 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-log-socket\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772651 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772678 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-cnibin\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772707 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772735 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-env-overrides\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.773530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772754 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-log-socket\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.774340 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772760 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a6da2f-baeb-425b-af20-9a71e6142b87-tmp\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.774340 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.772793 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.774340 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.772909 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:17.774340 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.772987 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret podName:31ad03fc-51f4-46b5-9a17-4d7f38513e84 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:18.272958379 +0000 UTC m=+2.029725625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret") pod "global-pull-secret-syncer-f62th" (UID: "31ad03fc-51f4-46b5-9a17-4d7f38513e84") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:17.774340 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.773463 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-env-overrides\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.775899 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.775748 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e7731ab-68de-44b8-8897-daeee9096d24-agent-certs\") pod \"konnectivity-agent-wvqr2\" (UID: \"6e7731ab-68de-44b8-8897-daeee9096d24\") " pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:17.775993 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.775811 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-ovn-node-metrics-cert\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.776856 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.776831 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6qng\" (UniqueName: \"kubernetes.io/projected/f8110c49-0824-494d-9f27-e0ba6e54a723-kube-api-access-x6qng\") pod \"iptables-alerter-crpps\" (UID: \"f8110c49-0824-494d-9f27-e0ba6e54a723\") " pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:17.781095 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.781070 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrk9r\" (UniqueName: \"kubernetes.io/projected/04add3b9-e936-4e41-9562-21031e041cc4-kube-api-access-nrk9r\") pod \"aws-ebs-csi-driver-node-thfhq\" (UID: \"04add3b9-e936-4e41-9562-21031e041cc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.781762 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.781739 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrjlf\" (UniqueName: \"kubernetes.io/projected/9c40e778-6932-421e-871c-7654d4d5912f-kube-api-access-hrjlf\") pod \"multus-xlplg\" (UID: \"9c40e778-6932-421e-871c-7654d4d5912f\") " pod="openshift-multus/multus-xlplg" Sep 17 01:37:17.783318 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.783296 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjmw\" (UniqueName: \"kubernetes.io/projected/4c2f9cad-e927-484e-9599-a9a84ade862e-kube-api-access-vhjmw\") pod \"node-resolver-k5mbc\" (UID: \"4c2f9cad-e927-484e-9599-a9a84ade862e\") " pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:17.783415 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.783355 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6x42\" (UniqueName: \"kubernetes.io/projected/d9b4c988-c58d-4e30-94e9-a892d64598b4-kube-api-access-v6x42\") pod \"node-ca-rf6rr\" (UID: \"d9b4c988-c58d-4e30-94e9-a892d64598b4\") " pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.784072 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.784052 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6f8\" (UniqueName: \"kubernetes.io/projected/cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8-kube-api-access-bm6f8\") pod \"ovnkube-node-rw7gk\" (UID: \"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:17.787977 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.787961 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rf6rr" Sep 17 01:37:17.873199 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873173 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-var-lib-kubelet\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.873199 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873202 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-kubernetes\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873220 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-cnibin\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873256 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a6da2f-baeb-425b-af20-9a71e6142b87-tmp\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873284 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-system-cni-dir\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873299 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-sys\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873299 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-var-lib-kubelet\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873308 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-kubernetes\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873333 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-tuned\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873353 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-cnibin\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873358 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873374 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-system-cni-dir\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873379 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-sysconfig\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873380 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-sys\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.873423 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873415 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-modprobe-d\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873443 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-sysconfig\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873458 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-run\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873494 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-sysctl-d\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873528 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-modprobe-d\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873532 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873590 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcm4f\" (UniqueName: \"kubernetes.io/projected/26801bb3-4b53-4677-a222-d71afd225350-kube-api-access-bcm4f\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873606 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-run\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.873618 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.873695 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs podName:26801bb3-4b53-4677-a222-d71afd225350 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:18.373675033 +0000 UTC m=+2.130442269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs") pod "network-metrics-daemon-bjxzr" (UID: "26801bb3-4b53-4677-a222-d71afd225350") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873705 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-sysctl-d\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873618 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873866 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kss5z\" (UniqueName: \"kubernetes.io/projected/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-kube-api-access-kss5z\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873899 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4vj\" (UniqueName: \"kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj\") pod \"network-check-target-rs9g5\" (UID: \"beb5923e-36ae-4f98-8dca-590530716c12\") " pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873924 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-sysctl-conf\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873934 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873948 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-host\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.874180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.873977 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-lib-modules\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874013 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874050 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874067 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-lib-modules\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874076 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dj6w\" (UniqueName: \"kubernetes.io/projected/05a6da2f-baeb-425b-af20-9a71e6142b87-kube-api-access-5dj6w\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874086 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874103 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-systemd\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874107 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-host\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874154 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-sysctl-conf\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874198 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-systemd\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874217 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.875040 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.874550 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.875764 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.875743 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a6da2f-baeb-425b-af20-9a71e6142b87-etc-tuned\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.875903 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.875889 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a6da2f-baeb-425b-af20-9a71e6142b87-tmp\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.880388 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.880372 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:17.880431 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.880398 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:17.880431 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.880412 2586 projected.go:196] Error preparing data for projected volume kube-api-access-lp4vj for pod openshift-network-diagnostics/network-check-target-rs9g5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:17.880494 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:17.880466 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj podName:beb5923e-36ae-4f98-8dca-590530716c12 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:18.380454533 +0000 UTC m=+2.137221773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4vj" (UniqueName: "kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj") pod "network-check-target-rs9g5" (UID: "beb5923e-36ae-4f98-8dca-590530716c12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:17.882249 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.882231 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dj6w\" (UniqueName: \"kubernetes.io/projected/05a6da2f-baeb-425b-af20-9a71e6142b87-kube-api-access-5dj6w\") pod \"tuned-9lpk2\" (UID: \"05a6da2f-baeb-425b-af20-9a71e6142b87\") " pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:17.882986 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.882970 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kss5z\" (UniqueName: \"kubernetes.io/projected/ba4e55ae-8f06-49da-af0c-18ded6ff0cd5-kube-api-access-kss5z\") pod \"multus-additional-cni-plugins-kxzh6\" (UID: \"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5\") " pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:17.883134 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.883118 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcm4f\" (UniqueName: \"kubernetes.io/projected/26801bb3-4b53-4677-a222-d71afd225350-kube-api-access-bcm4f\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:17.921701 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:17.921669 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ada5008fdb1ae2d02491334493ce88.slice/crio-25e488b84c68a63a64c60a8ba0ba0927be7aa789eb4f4131206acf53e400e4a2 WatchSource:0}: Error finding container 25e488b84c68a63a64c60a8ba0ba0927be7aa789eb4f4131206acf53e400e4a2: Status 404 returned error can't find the container with id 25e488b84c68a63a64c60a8ba0ba0927be7aa789eb4f4131206acf53e400e4a2 Sep 17 01:37:17.921938 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:17.921925 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5959b59db0da16a9835d45cb187dd20d.slice/crio-91172fa8a45950615f0a028850bcf2710f4f487a422c7c496ae055bad3000609 WatchSource:0}: Error finding container 91172fa8a45950615f0a028850bcf2710f4f487a422c7c496ae055bad3000609: Status 404 returned error can't find the container with id 91172fa8a45950615f0a028850bcf2710f4f487a422c7c496ae055bad3000609 Sep 17 01:37:17.926523 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.926506 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:37:17.975148 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.975115 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" Sep 17 01:37:17.984804 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:17.984785 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04add3b9_e936_4e41_9562_21031e041cc4.slice/crio-b926cb7fa5eea1d54c04adafee95ea56e41aa756650e375a2617cf9c01aef9aa WatchSource:0}: Error finding container b926cb7fa5eea1d54c04adafee95ea56e41aa756650e375a2617cf9c01aef9aa: Status 404 returned error can't find the container with id b926cb7fa5eea1d54c04adafee95ea56e41aa756650e375a2617cf9c01aef9aa Sep 17 01:37:17.995284 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:17.995265 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crpps" Sep 17 01:37:18.003648 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:18.003630 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8110c49_0824_494d_9f27_e0ba6e54a723.slice/crio-379d8733387f3a7ed36975ef76e1f4c2b2225458fde70b4cee86e6eb9f247431 WatchSource:0}: Error finding container 379d8733387f3a7ed36975ef76e1f4c2b2225458fde70b4cee86e6eb9f247431: Status 404 returned error can't find the container with id 379d8733387f3a7ed36975ef76e1f4c2b2225458fde70b4cee86e6eb9f247431 Sep 17 01:37:18.011697 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.011674 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:18.020865 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.020847 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xlplg" Sep 17 01:37:18.028350 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:18.028329 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c40e778_6932_421e_871c_7654d4d5912f.slice/crio-9a5731445b79733d322e04dff97d4cd13c6195076fae7d5ed384c38e7b052cb2 WatchSource:0}: Error finding container 9a5731445b79733d322e04dff97d4cd13c6195076fae7d5ed384c38e7b052cb2: Status 404 returned error can't find the container with id 9a5731445b79733d322e04dff97d4cd13c6195076fae7d5ed384c38e7b052cb2 Sep 17 01:37:18.042249 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.042221 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k5mbc" Sep 17 01:37:18.044124 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.044105 2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:18.047927 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.047911 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:18.050217 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:18.050194 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2f9cad_e927_484e_9599_a9a84ade862e.slice/crio-c558c578036452b1092643a87bd37ff59597662337d2f3d7f4047dedcce5fad4 WatchSource:0}: Error finding container c558c578036452b1092643a87bd37ff59597662337d2f3d7f4047dedcce5fad4: Status 404 returned error can't find the container with id c558c578036452b1092643a87bd37ff59597662337d2f3d7f4047dedcce5fad4 Sep 17 01:37:18.056799 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:18.056781 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8dd7d7_feaa_4fc6_abb1_b64abf73ffd8.slice/crio-824bc4ec80ce87721bdcbcdb82ae069f70b1bbbde51a94913e491bb5cecb6360 WatchSource:0}: Error finding container 824bc4ec80ce87721bdcbcdb82ae069f70b1bbbde51a94913e491bb5cecb6360: Status 404 returned error can't find the container with id 824bc4ec80ce87721bdcbcdb82ae069f70b1bbbde51a94913e491bb5cecb6360 Sep 17 01:37:18.100793 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.100752 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" Sep 17 01:37:18.107767 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.107747 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" Sep 17 01:37:18.115064 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:18.115035 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4e55ae_8f06_49da_af0c_18ded6ff0cd5.slice/crio-4c1e13fd1a60f30bf4aead0ba8754b67d322c5be4deb166912da0f87664209f1 WatchSource:0}: Error finding container 4c1e13fd1a60f30bf4aead0ba8754b67d322c5be4deb166912da0f87664209f1: Status 404 returned error can't find the container with id 4c1e13fd1a60f30bf4aead0ba8754b67d322c5be4deb166912da0f87664209f1 Sep 17 01:37:18.115709 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:18.115686 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b4c988_c58d_4e30_94e9_a892d64598b4.slice/crio-24ba9281842967672af8ea7b55c1731b02c826cc240d66b42c4367c598527a4c WatchSource:0}: Error finding container 24ba9281842967672af8ea7b55c1731b02c826cc240d66b42c4367c598527a4c: Status 404 returned error can't find the container with id 24ba9281842967672af8ea7b55c1731b02c826cc240d66b42c4367c598527a4c Sep 17 01:37:18.120630 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:18.120605 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a6da2f_baeb_425b_af20_9a71e6142b87.slice/crio-612729563d4cd9986094ecfe0518fb95b2c36e401130cc7366e4d290aeedc027 WatchSource:0}: Error finding container 612729563d4cd9986094ecfe0518fb95b2c36e401130cc7366e4d290aeedc027: Status 404 returned error can't find the container with id 612729563d4cd9986094ecfe0518fb95b2c36e401130cc7366e4d290aeedc027 Sep 17 01:37:18.125086 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.122684 2586 kuberuntime_manager.go:1459] "Unhandled Error" err="container tuned start failed in pod tuned-9lpk2_openshift-cluster-node-tuning-operator(05a6da2f-baeb-425b-af20-9a71e6142b87): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 17 01:37:18.125086 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.122738 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" podUID="05a6da2f-baeb-425b-af20-9a71e6142b87" Sep 17 01:37:18.277162 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.277080 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:18.277293 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.277221 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:18.277328 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.277300 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret podName:31ad03fc-51f4-46b5-9a17-4d7f38513e84 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:19.277285326 +0000 UTC m=+3.034052548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret") pod "global-pull-secret-syncer-f62th" (UID: "31ad03fc-51f4-46b5-9a17-4d7f38513e84") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:18.377351 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.377312 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:18.377495 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.377449 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:18.377576 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.377505 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs podName:26801bb3-4b53-4677-a222-d71afd225350 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:19.37748917 +0000 UTC m=+3.134256409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs") pod "network-metrics-daemon-bjxzr" (UID: "26801bb3-4b53-4677-a222-d71afd225350") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:18.480505 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.480472 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4vj\" (UniqueName: \"kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj\") pod \"network-check-target-rs9g5\" (UID: \"beb5923e-36ae-4f98-8dca-590530716c12\") " pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:18.480736 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.480720 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:18.480807 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.480746 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:18.480807 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.480761 2586 projected.go:196] Error preparing data for projected volume kube-api-access-lp4vj for pod openshift-network-diagnostics/network-check-target-rs9g5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:18.480913 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.480854 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj podName:beb5923e-36ae-4f98-8dca-590530716c12 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:19.480831284 +0000 UTC m=+3.237598525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4vj" (UniqueName: "kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj") pod "network-check-target-rs9g5" (UID: "beb5923e-36ae-4f98-8dca-590530716c12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:18.710096 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.709975 2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 01:32:18 +0000 UTC" deadline="2028-03-18 02:29:36.741269532 +0000 UTC" Sep 17 01:37:18.710096 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.710044 2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13152h52m18.031230887s" Sep 17 01:37:18.761895 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.761860 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:18.762076 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.762008 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:18.779830 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.779771 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" event={"ID":"05a6da2f-baeb-425b-af20-9a71e6142b87","Type":"ContainerStarted","Data":"612729563d4cd9986094ecfe0518fb95b2c36e401130cc7366e4d290aeedc027"} Sep 17 01:37:18.784800 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:18.784583 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" podUID="05a6da2f-baeb-425b-af20-9a71e6142b87" Sep 17 01:37:18.787395 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.787367 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rf6rr" event={"ID":"d9b4c988-c58d-4e30-94e9-a892d64598b4","Type":"ContainerStarted","Data":"24ba9281842967672af8ea7b55c1731b02c826cc240d66b42c4367c598527a4c"} Sep 17 01:37:18.794180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.794152 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" event={"ID":"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5","Type":"ContainerStarted","Data":"4c1e13fd1a60f30bf4aead0ba8754b67d322c5be4deb166912da0f87664209f1"} Sep 17 01:37:18.811545 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.811371 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" event={"ID":"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8","Type":"ContainerStarted","Data":"824bc4ec80ce87721bdcbcdb82ae069f70b1bbbde51a94913e491bb5cecb6360"} Sep 17 01:37:18.815774 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.815639 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k5mbc" event={"ID":"4c2f9cad-e927-484e-9599-a9a84ade862e","Type":"ContainerStarted","Data":"c558c578036452b1092643a87bd37ff59597662337d2f3d7f4047dedcce5fad4"} Sep 17 01:37:18.834168 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.834137 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" event={"ID":"5959b59db0da16a9835d45cb187dd20d","Type":"ContainerStarted","Data":"91172fa8a45950615f0a028850bcf2710f4f487a422c7c496ae055bad3000609"} Sep 17 01:37:18.841999 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.841968 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xlplg" event={"ID":"9c40e778-6932-421e-871c-7654d4d5912f","Type":"ContainerStarted","Data":"9a5731445b79733d322e04dff97d4cd13c6195076fae7d5ed384c38e7b052cb2"} Sep 17 01:37:18.845600 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.845571 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wvqr2" event={"ID":"6e7731ab-68de-44b8-8897-daeee9096d24","Type":"ContainerStarted","Data":"ee29626cff8d0529231b34eb71df48f5178a294450d5bfa534248ae5485de001"} Sep 17 01:37:18.849315 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.849289 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crpps" event={"ID":"f8110c49-0824-494d-9f27-e0ba6e54a723","Type":"ContainerStarted","Data":"379d8733387f3a7ed36975ef76e1f4c2b2225458fde70b4cee86e6eb9f247431"} Sep 17 01:37:18.856476 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.856443 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" event={"ID":"04add3b9-e936-4e41-9562-21031e041cc4","Type":"ContainerStarted","Data":"b926cb7fa5eea1d54c04adafee95ea56e41aa756650e375a2617cf9c01aef9aa"} Sep 17 01:37:18.861569 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:18.861543 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal" event={"ID":"65ada5008fdb1ae2d02491334493ce88","Type":"ContainerStarted","Data":"25e488b84c68a63a64c60a8ba0ba0927be7aa789eb4f4131206acf53e400e4a2"} Sep 17 01:37:19.012992 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:19.012871 2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:19.288481 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:19.287653 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:19.288481 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:19.287876 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:19.288481 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:19.287962 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret podName:31ad03fc-51f4-46b5-9a17-4d7f38513e84 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:21.287942852 +0000 UTC m=+5.044710082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret") pod "global-pull-secret-syncer-f62th" (UID: "31ad03fc-51f4-46b5-9a17-4d7f38513e84") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:19.388451 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:19.388410 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:19.388649 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:19.388593 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:19.388738 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:19.388681 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs podName:26801bb3-4b53-4677-a222-d71afd225350 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:21.38866021 +0000 UTC m=+5.145427435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs") pod "network-metrics-daemon-bjxzr" (UID: "26801bb3-4b53-4677-a222-d71afd225350") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:19.489689 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:19.489649 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4vj\" (UniqueName: \"kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj\") pod \"network-check-target-rs9g5\" (UID: \"beb5923e-36ae-4f98-8dca-590530716c12\") " pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:19.489889 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:19.489873 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:19.489962 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:19.489898 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:19.489962 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:19.489913 2586 projected.go:196] Error preparing data for projected volume kube-api-access-lp4vj for pod openshift-network-diagnostics/network-check-target-rs9g5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:19.490074 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:19.489977 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj podName:beb5923e-36ae-4f98-8dca-590530716c12 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:21.489956334 +0000 UTC m=+5.246723578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4vj" (UniqueName: "kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj") pod "network-check-target-rs9g5" (UID: "beb5923e-36ae-4f98-8dca-590530716c12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:19.561869 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:19.561792 2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:19.710865 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:19.710817 2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 01:32:18 +0000 UTC" deadline="2028-03-28 05:29:24.673334926 +0000 UTC" Sep 17 01:37:19.710865 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:19.710862 2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13395h52m4.962477539s" Sep 17 01:37:19.761988 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:19.761958 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:19.762277 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:19.762097 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:19.762595 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:19.762518 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:19.762741 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:19.762622 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:20.764022 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:20.763417 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:20.764022 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:20.763555 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:21.307940 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:21.307895 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:21.308132 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:21.308076 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:21.308213 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:21.308143 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret podName:31ad03fc-51f4-46b5-9a17-4d7f38513e84 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:25.308122651 +0000 UTC m=+9.064889889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret") pod "global-pull-secret-syncer-f62th" (UID: "31ad03fc-51f4-46b5-9a17-4d7f38513e84") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:21.409382 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:21.409340 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:21.409566 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:21.409485 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:21.409566 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:21.409551 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs podName:26801bb3-4b53-4677-a222-d71afd225350 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:25.409533308 +0000 UTC m=+9.166300536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs") pod "network-metrics-daemon-bjxzr" (UID: "26801bb3-4b53-4677-a222-d71afd225350") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:21.510748 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:21.510696 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4vj\" (UniqueName: \"kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj\") pod \"network-check-target-rs9g5\" (UID: \"beb5923e-36ae-4f98-8dca-590530716c12\") " pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:21.510961 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:21.510888 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:21.510961 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:21.510929 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:21.510961 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:21.510941 2586 projected.go:196] Error preparing data for projected volume kube-api-access-lp4vj for pod openshift-network-diagnostics/network-check-target-rs9g5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:21.511090 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:21.510995 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj podName:beb5923e-36ae-4f98-8dca-590530716c12 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:25.510975856 +0000 UTC m=+9.267743093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4vj" (UniqueName: "kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj") pod "network-check-target-rs9g5" (UID: "beb5923e-36ae-4f98-8dca-590530716c12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:21.761988 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:21.761303 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:21.761988 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:21.761447 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:21.762424 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:21.762393 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:21.762538 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:21.762501 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:22.761438 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:22.761397 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:22.761880 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:22.761555 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:23.762068 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:23.762029 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:23.762494 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:23.762181 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:23.762602 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:23.762582 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:23.762729 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:23.762706 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:24.761162 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:24.761124 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:24.761328 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:24.761282 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:25.344003 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:25.343939 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:25.344564 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:25.344063 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:25.344564 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:25.344126 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret podName:31ad03fc-51f4-46b5-9a17-4d7f38513e84 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:33.344113432 +0000 UTC m=+17.100880659 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret") pod "global-pull-secret-syncer-f62th" (UID: "31ad03fc-51f4-46b5-9a17-4d7f38513e84") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:25.445153 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:25.445118 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:25.445463 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:25.445274 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:25.445463 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:25.445343 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs podName:26801bb3-4b53-4677-a222-d71afd225350 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:33.445324975 +0000 UTC m=+17.202092202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs") pod "network-metrics-daemon-bjxzr" (UID: "26801bb3-4b53-4677-a222-d71afd225350") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:25.546019 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:25.545912 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4vj\" (UniqueName: \"kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj\") pod \"network-check-target-rs9g5\" (UID: \"beb5923e-36ae-4f98-8dca-590530716c12\") " pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:25.546206 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:25.546138 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:25.546206 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:25.546166 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:25.546206 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:25.546180 2586 projected.go:196] Error preparing data for projected volume kube-api-access-lp4vj for pod openshift-network-diagnostics/network-check-target-rs9g5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:25.546342 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:25.546245 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj podName:beb5923e-36ae-4f98-8dca-590530716c12 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:33.546223358 +0000 UTC m=+17.302990588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4vj" (UniqueName: "kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj") pod "network-check-target-rs9g5" (UID: "beb5923e-36ae-4f98-8dca-590530716c12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:25.763116 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:25.762710 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:25.763116 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:25.762881 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:25.763116 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:25.762923 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:25.763116 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:25.763033 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:26.762304 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:26.762259 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:26.762726 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:26.762399 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:27.761684 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:27.761022 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:27.761684 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:27.761184 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:27.761684 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:27.761516 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:27.761684 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:27.761617 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:27.912296 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:27.912103 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal" event={"ID":"65ada5008fdb1ae2d02491334493ce88","Type":"ContainerStarted","Data":"b60f4464411d96cb2d4984ad36e01204a9eee3e56ee7f792994b8b91ce926e56"} Sep 17 01:37:28.761948 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.761916 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:28.762138 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:28.762036 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:28.915866 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.915821 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rf6rr" event={"ID":"d9b4c988-c58d-4e30-94e9-a892d64598b4","Type":"ContainerStarted","Data":"f3d17a4e64f1e48c40beb263a89a0a351d54299016df0c2c88c5903aec1e4c7a"} Sep 17 01:37:28.918199 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.918160 2586 generic.go:356] "Generic (PLEG): container finished" podID="ba4e55ae-8f06-49da-af0c-18ded6ff0cd5" containerID="f548e2ebd1b6de789f5e095073b0e0928a84822133e7c730b322216df72c87ae" exitCode=0 Sep 17 01:37:28.918331 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.918224 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" event={"ID":"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5","Type":"ContainerDied","Data":"f548e2ebd1b6de789f5e095073b0e0928a84822133e7c730b322216df72c87ae"} Sep 17 01:37:28.920871 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.920845 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k5mbc" event={"ID":"4c2f9cad-e927-484e-9599-a9a84ade862e","Type":"ContainerStarted","Data":"d02374f2966811dbbcf2721d0afd9e7cce3af3b6334ceec69b9a56ad06f66ee4"} Sep 17 01:37:28.922878 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.922850 2586 generic.go:356] "Generic (PLEG): container finished" podID="5959b59db0da16a9835d45cb187dd20d" containerID="f1fae95a362384747684696dcb0865e38f944e986f6eebf03a8459edd4b2b6a0" exitCode=0 Sep 17 01:37:28.922878 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.922890 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" event={"ID":"5959b59db0da16a9835d45cb187dd20d","Type":"ContainerDied","Data":"f1fae95a362384747684696dcb0865e38f944e986f6eebf03a8459edd4b2b6a0"} Sep 17 01:37:28.924720 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.924695 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wvqr2" event={"ID":"6e7731ab-68de-44b8-8897-daeee9096d24","Type":"ContainerStarted","Data":"2bdd34b4dedb43fdb4578a356aa72603ea4da4c15637fc714ee763aa8ca291a4"} Sep 17 01:37:28.926418 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.926394 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" event={"ID":"04add3b9-e936-4e41-9562-21031e041cc4","Type":"ContainerStarted","Data":"4f9c219a01649275af43490f5a2c3ceaecdecd80cbae84aa0cb4276bb23e887f"} Sep 17 01:37:28.929538 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.929491 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-174.ec2.internal" podStartSLOduration=10.929474138 podStartE2EDuration="10.929474138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:27.927516594 +0000 UTC m=+11.684283839" watchObservedRunningTime="2026-09-17 01:37:28.929474138 +0000 UTC m=+12.686241381" Sep 17 01:37:28.930102 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.930068 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rf6rr" podStartSLOduration=2.496690951 podStartE2EDuration="11.930058506s" totalImagesPullingTime="9.433367555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:28.929627463 +0000 UTC m=+12.686394685" watchObservedRunningTime="2026-09-17 01:37:28.930058506 +0000 UTC m=+12.686825749" Sep 17 01:37:28.986612 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.986531 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k5mbc" podStartSLOduration=2.4812344250000002 podStartE2EDuration="11.986486824s" totalImagesPullingTime="9.505252399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:28.962010314 +0000 UTC m=+12.718777583" watchObservedRunningTime="2026-09-17 01:37:28.986486824 +0000 UTC m=+12.743254068" Sep 17 01:37:28.995794 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:28.995743 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wvqr2" podStartSLOduration=2.473964415 podStartE2EDuration="11.995727106s" totalImagesPullingTime="9.521762691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:28.995724294 +0000 UTC m=+12.752491564" watchObservedRunningTime="2026-09-17 01:37:28.995727106 +0000 UTC m=+12.752494341" Sep 17 01:37:29.761372 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:29.761339 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:29.761372 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:29.761365 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:29.761608 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:29.761461 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:29.761608 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:29.761529 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:29.930269 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:29.930219 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crpps" event={"ID":"f8110c49-0824-494d-9f27-e0ba6e54a723","Type":"ContainerStarted","Data":"daced3142b24ec26a8256f6de575af6880a0196e532ad028c22e46547c816fda"} Sep 17 01:37:29.945360 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:29.945296 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-crpps" podStartSLOduration=3.393440371 podStartE2EDuration="12.94526002s" totalImagesPullingTime="9.551819649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:29.944712838 +0000 UTC m=+13.701480107" watchObservedRunningTime="2026-09-17 01:37:29.94526002 +0000 UTC m=+13.702027263" Sep 17 01:37:30.761697 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:30.761647 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:30.761930 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:30.761899 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:31.762006 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:31.761968 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:31.762480 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:31.762021 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:31.762480 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:31.762097 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:31.762480 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:31.762467 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:32.761444 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:32.761408 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:32.761757 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:32.761534 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:33.158304 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:33.158269 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:33.158783 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:33.158692 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:33.409946 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:33.409862 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:33.410191 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:33.410049 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:33.410261 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:33.410208 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret podName:31ad03fc-51f4-46b5-9a17-4d7f38513e84 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:49.410189222 +0000 UTC m=+33.166956459 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret") pod "global-pull-secret-syncer-f62th" (UID: "31ad03fc-51f4-46b5-9a17-4d7f38513e84") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:33.510958 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:33.510906 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:33.511148 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:33.511120 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:33.511213 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:33.511182 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs podName:26801bb3-4b53-4677-a222-d71afd225350 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:49.511163458 +0000 UTC m=+33.267930694 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs") pod "network-metrics-daemon-bjxzr" (UID: "26801bb3-4b53-4677-a222-d71afd225350") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:33.612193 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:33.612155 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4vj\" (UniqueName: \"kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj\") pod \"network-check-target-rs9g5\" (UID: \"beb5923e-36ae-4f98-8dca-590530716c12\") " pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:33.612377 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:33.612335 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:33.612377 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:33.612361 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:33.612377 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:33.612374 2586 projected.go:196] Error preparing data for projected volume kube-api-access-lp4vj for pod openshift-network-diagnostics/network-check-target-rs9g5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:33.612543 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:33.612444 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj podName:beb5923e-36ae-4f98-8dca-590530716c12 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:49.612423958 +0000 UTC m=+33.369191186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4vj" (UniqueName: "kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj") pod "network-check-target-rs9g5" (UID: "beb5923e-36ae-4f98-8dca-590530716c12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:33.761401 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:33.761321 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:33.761565 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:33.761469 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:33.761906 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:33.761887 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:33.762004 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:33.761984 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:34.761839 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:34.761805 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:34.762285 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:34.761932 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:35.762018 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:35.761982 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:35.762471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:35.762038 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:35.762471 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:35.762125 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:35.762471 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:35.762257 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:36.762105 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:36.761910 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:36.762491 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:36.762171 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:37.761715 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:37.761674 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:37.761899 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:37.761675 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:37.761899 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:37.761809 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:37.762017 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:37.761909 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:38.762945 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:38.762757 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:38.762945 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:38.762889 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:39.761240 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:39.761197 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:39.761240 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:39.761238 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:39.761595 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:39.761330 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:39.761595 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:39.761417 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:40.763794 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.763756 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:40.764687 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:40.763877 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:40.817378 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.817139 2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 01:37:40.954777 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.954659 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xlplg" event={"ID":"9c40e778-6932-421e-871c-7654d4d5912f","Type":"ContainerStarted","Data":"1ed474cc99997f7700d1f8b0ece2364e1b4caa1f5ffd77f862032a35d2c35086"} Sep 17 01:37:40.956852 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.956826 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" event={"ID":"04add3b9-e936-4e41-9562-21031e041cc4","Type":"ContainerStarted","Data":"ed04afa7a9c34542b18446376dcdfb66de383027a6a7082535e108674296d08f"} Sep 17 01:37:40.958511 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.958482 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" event={"ID":"05a6da2f-baeb-425b-af20-9a71e6142b87","Type":"ContainerStarted","Data":"167b715d73dfdf86704999b739d1ce422e3d1bbfc0a7c333aab0087872179054"} Sep 17 01:37:40.960413 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.960382 2586 generic.go:356] "Generic (PLEG): container finished" podID="ba4e55ae-8f06-49da-af0c-18ded6ff0cd5" containerID="90090ee0f736bd230f41cef5b88090ea4b28171a83dce019120e758912543091" exitCode=0 Sep 17 01:37:40.960517 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.960441 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" event={"ID":"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5","Type":"ContainerDied","Data":"90090ee0f736bd230f41cef5b88090ea4b28171a83dce019120e758912543091"} Sep 17 01:37:40.963039 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.963013 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" event={"ID":"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8","Type":"ContainerStarted","Data":"98b22341519e3b86d4cbe48ec1050c681eea4210be0ac800ca8851bfbce47420"} Sep 17 01:37:40.963136 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.963080 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" event={"ID":"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8","Type":"ContainerStarted","Data":"397269b95576055ffacbfd11bf839b3dec178e58f3eb3daed163a36fecdb00aa"} Sep 17 01:37:40.963136 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.963111 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" event={"ID":"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8","Type":"ContainerStarted","Data":"c8da27cfc4dc43959f5733482b6b8a55d3ddd1f9aeac4cb07a229237918de206"} Sep 17 01:37:40.965156 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.965133 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" event={"ID":"5959b59db0da16a9835d45cb187dd20d","Type":"ContainerStarted","Data":"73af71927a5342ffce5dd92e8d8f328c80895cfae6c0764b8fc8dc3925c4fb9d"} Sep 17 01:37:40.970868 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:40.970817 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xlplg" podStartSLOduration=1.39846348 podStartE2EDuration="23.970799064s" totalImagesPullingTime="22.572335584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:40.969684063 +0000 UTC m=+24.726451322" watchObservedRunningTime="2026-09-17 01:37:40.970799064 +0000 UTC m=+24.727566309" Sep 17 01:37:41.007236 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:41.007194 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-174.ec2.internal" podStartSLOduration=23.007177413 podStartE2EDuration="23.007177413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:40.987472498 +0000 UTC m=+24.744239742" watchObservedRunningTime="2026-09-17 01:37:41.007177413 +0000 UTC m=+24.763944656" Sep 17 01:37:41.731612 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:41.731482 2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T01:37:40.817359341Z","UUID":"a989e5a9-d330-4312-b854-de26ea92e8e9","Handler":null,"Name":"","Endpoint":""} Sep 17 01:37:41.732972 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:41.732945 2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 01:37:41.733159 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:41.732982 2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 01:37:41.761100 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:41.761061 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:41.761292 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:41.761078 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:41.761292 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:41.761197 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:41.761292 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:41.761274 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:41.970910 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:41.970880 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" event={"ID":"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8","Type":"ContainerStarted","Data":"1486d7d04eb975dcf286f8f1a886b0088923fd70754e1901d15377cbe1727928"} Sep 17 01:37:41.971284 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:41.970970 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" event={"ID":"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8","Type":"ContainerStarted","Data":"4bfe33ef7947879ad2f9725bd23f466a39f5d33bd764571a064830fd012c2c84"} Sep 17 01:37:41.971284 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:41.971030 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" event={"ID":"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8","Type":"ContainerStarted","Data":"0bb81117485450efac5e1ef5337c36f164ab0c69c273ae451983a64f100c5683"} Sep 17 01:37:42.761385 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:42.761344 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:42.761553 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:42.761449 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:42.975328 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:42.975297 2586 generic.go:356] "Generic (PLEG): container finished" podID="ba4e55ae-8f06-49da-af0c-18ded6ff0cd5" containerID="a86b6a587115a9f2064fa29551b802aafd5a1cb3dbada990dd843408ebd651bb" exitCode=0 Sep 17 01:37:42.975782 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:42.975376 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" event={"ID":"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5","Type":"ContainerDied","Data":"a86b6a587115a9f2064fa29551b802aafd5a1cb3dbada990dd843408ebd651bb"} Sep 17 01:37:42.977535 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:42.977509 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" event={"ID":"04add3b9-e936-4e41-9562-21031e041cc4","Type":"ContainerStarted","Data":"c1028393427a9b352e9d9bda120967a93badc54cf1ef88cb70a11ee6f363cca7"} Sep 17 01:37:43.002200 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:43.002147 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9lpk2" podStartSLOduration=3.4796761370000002 podStartE2EDuration="26.00213133s" totalImagesPullingTime="22.522455193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-17 01:37:41.022159135 +0000 UTC m=+24.778926375" watchObservedRunningTime="2026-09-17 01:37:43.00213133 +0000 UTC m=+26.758898574" Sep 17 01:37:43.147115 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:43.147081 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:43.147260 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:43.147247 2586 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 17 01:37:43.147904 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:43.147880 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wvqr2" Sep 17 01:37:43.163079 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:43.163040 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thfhq" podStartSLOduration=2.80692342 podStartE2EDuration="26.16302379s" totalImagesPullingTime="23.35610037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:43.023752937 +0000 UTC m=+26.780520180" watchObservedRunningTime="2026-09-17 01:37:43.16302379 +0000 UTC m=+26.919791035" Sep 17 01:37:43.761626 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:43.761563 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:43.761786 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:43.761693 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:43.761786 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:43.761749 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:43.761867 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:43.761844 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:43.982508 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:43.982476 2586 generic.go:356] "Generic (PLEG): container finished" podID="ba4e55ae-8f06-49da-af0c-18ded6ff0cd5" containerID="f7c419edc9fd124f0e5264515d17d44907956e6d302a5e97bb36d08294c2070d" exitCode=0 Sep 17 01:37:43.982901 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:43.982553 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" event={"ID":"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5","Type":"ContainerDied","Data":"f7c419edc9fd124f0e5264515d17d44907956e6d302a5e97bb36d08294c2070d"} Sep 17 01:37:43.985909 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:43.985886 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" event={"ID":"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8","Type":"ContainerStarted","Data":"4fcd5a9b0cb1fd7e7a0533566bf1d21a103dc7de91e8e84c1a368059ea3c50eb"} Sep 17 01:37:44.761732 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:44.761692 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:44.761918 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:44.761808 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:45.761687 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:45.761419 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:45.762223 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:45.761461 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:45.762223 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:45.761715 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:45.762223 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:45.761770 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:45.996697 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:45.996656 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" event={"ID":"cf8dd7d7-feaa-4fc6-abb1-b64abf73ffd8","Type":"ContainerStarted","Data":"ee6275baf95054983cdff9dea5013469e943093aa59d8dd39f9fc9c7f7454a7c"} Sep 17 01:37:45.997110 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:45.997021 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:46.013896 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:46.013828 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:46.024427 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:46.024372 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" podStartSLOduration=6.458764187 podStartE2EDuration="29.02435776s" totalImagesPullingTime="22.565593573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:46.023156445 +0000 UTC m=+29.779923707" watchObservedRunningTime="2026-09-17 01:37:46.02435776 +0000 UTC m=+29.781125003" Sep 17 01:37:46.763837 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:46.763804 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:46.766513 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:46.764209 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:46.999877 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:46.999845 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:47.000036 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:46.999935 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:47.018053 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:47.017966 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:37:47.700286 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:47.700230 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f62th"] Sep 17 01:37:47.700492 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:47.700371 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:47.700492 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:47.700475 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:47.704691 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:47.704231 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rs9g5"] Sep 17 01:37:47.704691 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:47.704297 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bjxzr"] Sep 17 01:37:47.704691 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:47.704400 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:47.704691 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:47.704503 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:47.704691 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:47.704594 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:47.704691 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:47.704657 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:48.761748 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:48.761704 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:48.762406 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:48.761825 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:49.418809 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:49.418762 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:49.419005 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:49.418988 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:49.419074 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:49.419063 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret podName:31ad03fc-51f4-46b5-9a17-4d7f38513e84 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:21.419045776 +0000 UTC m=+65.175813035 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret") pod "global-pull-secret-syncer-f62th" (UID: "31ad03fc-51f4-46b5-9a17-4d7f38513e84") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:49.520095 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:49.519915 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:49.520231 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:49.520061 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:49.520231 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:49.520158 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs podName:26801bb3-4b53-4677-a222-d71afd225350 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:21.520143528 +0000 UTC m=+65.276910762 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs") pod "network-metrics-daemon-bjxzr" (UID: "26801bb3-4b53-4677-a222-d71afd225350") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:49.621179 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:49.621145 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4vj\" (UniqueName: \"kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj\") pod \"network-check-target-rs9g5\" (UID: \"beb5923e-36ae-4f98-8dca-590530716c12\") " pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:49.621328 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:49.621308 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:49.621391 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:49.621332 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:49.621391 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:49.621343 2586 projected.go:196] Error preparing data for projected volume kube-api-access-lp4vj for pod openshift-network-diagnostics/network-check-target-rs9g5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:49.621472 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:49.621392 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj podName:beb5923e-36ae-4f98-8dca-590530716c12 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:21.621378398 +0000 UTC m=+65.378145632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4vj" (UniqueName: "kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj") pod "network-check-target-rs9g5" (UID: "beb5923e-36ae-4f98-8dca-590530716c12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:49.761957 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:49.761856 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:49.761957 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:49.761868 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:49.762322 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:49.761986 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:49.762322 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:49.762024 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:50.008938 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:50.008906 2586 generic.go:356] "Generic (PLEG): container finished" podID="ba4e55ae-8f06-49da-af0c-18ded6ff0cd5" containerID="e082f16c9da3f8eb3af095f5695a96470e477860dad7b0ee7dc7a9c246a1a27b" exitCode=0 Sep 17 01:37:50.009137 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:50.008962 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" event={"ID":"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5","Type":"ContainerDied","Data":"e082f16c9da3f8eb3af095f5695a96470e477860dad7b0ee7dc7a9c246a1a27b"} Sep 17 01:37:50.761236 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:50.761200 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:50.761404 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:50.761338 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:51.014234 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:51.014156 2586 generic.go:356] "Generic (PLEG): container finished" podID="ba4e55ae-8f06-49da-af0c-18ded6ff0cd5" containerID="b62bc40b92f9bdb0153922eafabe8eb2266128bdf5acafab4ca4efb12de23be8" exitCode=0 Sep 17 01:37:51.014234 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:51.014219 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" event={"ID":"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5","Type":"ContainerDied","Data":"b62bc40b92f9bdb0153922eafabe8eb2266128bdf5acafab4ca4efb12de23be8"} Sep 17 01:37:51.761092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:51.761056 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:51.761284 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:51.761160 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:37:51.761284 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:51.761223 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:51.761391 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:51.761331 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f62th" podUID="31ad03fc-51f4-46b5-9a17-4d7f38513e84" Sep 17 01:37:52.020437 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:52.020360 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" event={"ID":"ba4e55ae-8f06-49da-af0c-18ded6ff0cd5","Type":"ContainerStarted","Data":"9a88be8142802406a83d88587c7d56b107dda1765583eb01674395218373ef74"} Sep 17 01:37:52.040527 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:52.040478 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kxzh6" podStartSLOduration=6.886018688 podStartE2EDuration="35.040464275s" totalImagesPullingTime="28.154445587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:52.039099699 +0000 UTC m=+35.795866956" watchObservedRunningTime="2026-09-17 01:37:52.040464275 +0000 UTC m=+35.797231519" Sep 17 01:37:52.761508 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:52.761474 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:52.761765 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:52.761570 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs9g5" podUID="beb5923e-36ae-4f98-8dca-590530716c12" Sep 17 01:37:53.564834 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.564805 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-174.ec2.internal" event="NodeReady" Sep 17 01:37:53.565211 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.564918 2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 01:37:53.599532 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.599447 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb"] Sep 17 01:37:53.621537 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.621504 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c6587d777-q8lc4"] Sep 17 01:37:53.621704 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.621689 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:37:53.623839 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.623815 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 17 01:37:53.624295 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.624101 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gjn79\"" Sep 17 01:37:53.624403 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.624206 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 17 01:37:53.645812 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.645777 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb"] Sep 17 01:37:53.645812 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.645816 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c6587d777-q8lc4"] Sep 17 01:37:53.646157 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.645832 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2djzb"] Sep 17 01:37:53.646157 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.645901 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.648166 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.648142 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 01:37:53.648273 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.648142 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 01:37:53.648335 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.648322 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7d5d6\"" Sep 17 01:37:53.648443 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.648424 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 01:37:53.653543 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.653520 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 01:37:53.665750 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.665720 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pb4gq"] Sep 17 01:37:53.665863 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.665846 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2djzb" Sep 17 01:37:53.668852 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.668828 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 01:37:53.668987 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.668925 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 01:37:53.669075 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.669056 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gfnxn\"" Sep 17 01:37:53.669205 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.669188 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 01:37:53.685870 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.685833 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2djzb"] Sep 17 01:37:53.686072 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.685891 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pb4gq"] Sep 17 01:37:53.686072 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.685979 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.688295 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.688271 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 01:37:53.688406 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.688273 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 01:37:53.688406 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.688308 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-crsck\"" Sep 17 01:37:53.748304 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748270 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.748471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748319 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac42c232-24c7-4609-a169-07fe9bb4c2e6-tmp-dir\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.748471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748343 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.748471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748372 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8ed7b30-477c-480e-888f-94a13fb9892d-ca-trust-extracted\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.748471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748387 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-trusted-ca\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.748471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748402 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmcb\" (UniqueName: \"kubernetes.io/projected/3ce012e5-f6f2-4313-9085-430520d3d06d-kube-api-access-5hmcb\") pod \"ingress-canary-2djzb\" (UID: \"3ce012e5-f6f2-4313-9085-430520d3d06d\") " pod="openshift-ingress-canary/ingress-canary-2djzb" Sep 17 01:37:53.748471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748417 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk28f\" (UniqueName: \"kubernetes.io/projected/ac42c232-24c7-4609-a169-07fe9bb4c2e6-kube-api-access-gk28f\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.748471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748432 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-installation-pull-secrets\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.748711 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748490 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ce012e5-f6f2-4313-9085-430520d3d06d-cert\") pod \"ingress-canary-2djzb\" (UID: \"3ce012e5-f6f2-4313-9085-430520d3d06d\") " pod="openshift-ingress-canary/ingress-canary-2djzb" Sep 17 01:37:53.748711 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748555 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-bound-sa-token\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.748711 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748618 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:37:53.748711 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748649 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-image-registry-private-configuration\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.748711 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748671 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwlsr\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-kube-api-access-rwlsr\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.748711 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748706 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5467c639-5928-4a5f-a109-18c20b43fcfa-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:37:53.748886 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748738 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac42c232-24c7-4609-a169-07fe9bb4c2e6-config-volume\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.748886 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.748755 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-certificates\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.761419 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.761393 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:37:53.761419 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.761409 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:37:53.763808 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.763788 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 01:37:53.764052 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.764039 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fdmc\"" Sep 17 01:37:53.764118 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.764094 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 01:37:53.849722 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.849628 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-bound-sa-token\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.849722 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.849688 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:37:53.849722 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.849714 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-image-registry-private-configuration\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.849988 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.849736 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwlsr\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-kube-api-access-rwlsr\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.849988 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.849763 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5467c639-5928-4a5f-a109-18c20b43fcfa-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:37:53.849988 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.849790 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac42c232-24c7-4609-a169-07fe9bb4c2e6-config-volume\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.849988 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:53.849874 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 01:37:53.849988 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:53.849947 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert podName:5467c639-5928-4a5f-a109-18c20b43fcfa nodeName:}" failed. No retries permitted until 2026-09-17 01:37:54.349931795 +0000 UTC m=+38.106699017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-9r2gb" (UID: "5467c639-5928-4a5f-a109-18c20b43fcfa") : secret "networking-console-plugin-cert" not found Sep 17 01:37:53.850275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.849998 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-certificates\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.850275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850075 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.850275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850113 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac42c232-24c7-4609-a169-07fe9bb4c2e6-tmp-dir\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.850275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850142 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.850275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850162 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8ed7b30-477c-480e-888f-94a13fb9892d-ca-trust-extracted\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.850275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850182 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-trusted-ca\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.850275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850207 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmcb\" (UniqueName: \"kubernetes.io/projected/3ce012e5-f6f2-4313-9085-430520d3d06d-kube-api-access-5hmcb\") pod \"ingress-canary-2djzb\" (UID: \"3ce012e5-f6f2-4313-9085-430520d3d06d\") " pod="openshift-ingress-canary/ingress-canary-2djzb" Sep 17 01:37:53.850275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850233 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk28f\" (UniqueName: \"kubernetes.io/projected/ac42c232-24c7-4609-a169-07fe9bb4c2e6-kube-api-access-gk28f\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.850275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850254 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-installation-pull-secrets\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.850275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850277 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ce012e5-f6f2-4313-9085-430520d3d06d-cert\") pod \"ingress-canary-2djzb\" (UID: \"3ce012e5-f6f2-4313-9085-430520d3d06d\") " pod="openshift-ingress-canary/ingress-canary-2djzb" Sep 17 01:37:53.850787 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850431 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac42c232-24c7-4609-a169-07fe9bb4c2e6-tmp-dir\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.850787 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850469 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac42c232-24c7-4609-a169-07fe9bb4c2e6-config-volume\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.850787 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:53.850695 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:37:53.850787 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:53.850712 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c6587d777-q8lc4: secret "image-registry-tls" not found Sep 17 01:37:53.850787 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.850737 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-certificates\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.850787 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:53.850758 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls podName:d8ed7b30-477c-480e-888f-94a13fb9892d nodeName:}" failed. No retries permitted until 2026-09-17 01:37:54.350742577 +0000 UTC m=+38.107509816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls") pod "image-registry-6c6587d777-q8lc4" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d") : secret "image-registry-tls" not found Sep 17 01:37:53.851058 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:53.850849 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:37:53.851058 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:53.850913 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls podName:ac42c232-24c7-4609-a169-07fe9bb4c2e6 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:54.350896593 +0000 UTC m=+38.107663829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls") pod "dns-default-pb4gq" (UID: "ac42c232-24c7-4609-a169-07fe9bb4c2e6") : secret "dns-default-metrics-tls" not found Sep 17 01:37:53.851058 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.851011 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8ed7b30-477c-480e-888f-94a13fb9892d-ca-trust-extracted\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.851234 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.851195 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-trusted-ca\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.854227 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.854204 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-installation-pull-secrets\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.854385 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.854365 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ce012e5-f6f2-4313-9085-430520d3d06d-cert\") pod \"ingress-canary-2djzb\" (UID: \"3ce012e5-f6f2-4313-9085-430520d3d06d\") " pod="openshift-ingress-canary/ingress-canary-2djzb" Sep 17 01:37:53.854678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.854659 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-image-registry-private-configuration\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.858667 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.858640 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-bound-sa-token\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.858957 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.858933 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwlsr\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-kube-api-access-rwlsr\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:53.859281 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.859262 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk28f\" (UniqueName: \"kubernetes.io/projected/ac42c232-24c7-4609-a169-07fe9bb4c2e6-kube-api-access-gk28f\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:53.859919 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.859896 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmcb\" (UniqueName: \"kubernetes.io/projected/3ce012e5-f6f2-4313-9085-430520d3d06d-kube-api-access-5hmcb\") pod \"ingress-canary-2djzb\" (UID: \"3ce012e5-f6f2-4313-9085-430520d3d06d\") " pod="openshift-ingress-canary/ingress-canary-2djzb" Sep 17 01:37:53.867961 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.867935 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5467c639-5928-4a5f-a109-18c20b43fcfa-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:37:53.974677 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:53.974639 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2djzb" Sep 17 01:37:54.261248 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:54.261217 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2djzb"] Sep 17 01:37:54.264869 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:37:54.264839 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce012e5_f6f2_4313_9085_430520d3d06d.slice/crio-922105b6dc295abffb088512f1eeaa451efd6c3f25b8541b955ccfc9df09b055 WatchSource:0}: Error finding container 922105b6dc295abffb088512f1eeaa451efd6c3f25b8541b955ccfc9df09b055: Status 404 returned error can't find the container with id 922105b6dc295abffb088512f1eeaa451efd6c3f25b8541b955ccfc9df09b055 Sep 17 01:37:54.355409 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:54.355363 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:37:54.355643 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:54.355433 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:54.355643 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:54.355467 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:54.355643 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:54.355515 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 01:37:54.355643 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:54.355566 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:37:54.355643 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:54.355600 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:37:54.355643 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:54.355624 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert podName:5467c639-5928-4a5f-a109-18c20b43fcfa nodeName:}" failed. No retries permitted until 2026-09-17 01:37:55.355601496 +0000 UTC m=+39.112368733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-9r2gb" (UID: "5467c639-5928-4a5f-a109-18c20b43fcfa") : secret "networking-console-plugin-cert" not found Sep 17 01:37:54.355886 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:54.355656 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls podName:ac42c232-24c7-4609-a169-07fe9bb4c2e6 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:55.355640069 +0000 UTC m=+39.112407296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls") pod "dns-default-pb4gq" (UID: "ac42c232-24c7-4609-a169-07fe9bb4c2e6") : secret "dns-default-metrics-tls" not found Sep 17 01:37:54.355886 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:54.355602 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c6587d777-q8lc4: secret "image-registry-tls" not found Sep 17 01:37:54.355886 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:54.355701 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls podName:d8ed7b30-477c-480e-888f-94a13fb9892d nodeName:}" failed. No retries permitted until 2026-09-17 01:37:55.355688846 +0000 UTC m=+39.112456069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls") pod "image-registry-6c6587d777-q8lc4" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d") : secret "image-registry-tls" not found Sep 17 01:37:54.761708 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:54.761675 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:37:54.765913 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:54.764330 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 01:37:54.765913 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:54.764396 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 01:37:54.765913 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:54.764330 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hdsht\"" Sep 17 01:37:55.028695 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:55.028606 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2djzb" event={"ID":"3ce012e5-f6f2-4313-9085-430520d3d06d","Type":"ContainerStarted","Data":"922105b6dc295abffb088512f1eeaa451efd6c3f25b8541b955ccfc9df09b055"} Sep 17 01:37:55.363515 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:55.363481 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:55.363739 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:55.363557 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:55.363739 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:55.363648 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:37:55.363739 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:55.363670 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:37:55.363739 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:55.363701 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:37:55.363739 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:55.363719 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c6587d777-q8lc4: secret "image-registry-tls" not found Sep 17 01:37:55.363984 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:55.363753 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls podName:ac42c232-24c7-4609-a169-07fe9bb4c2e6 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:57.363731245 +0000 UTC m=+41.120498468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls") pod "dns-default-pb4gq" (UID: "ac42c232-24c7-4609-a169-07fe9bb4c2e6") : secret "dns-default-metrics-tls" not found Sep 17 01:37:55.363984 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:55.363763 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 01:37:55.363984 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:55.363782 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls podName:d8ed7b30-477c-480e-888f-94a13fb9892d nodeName:}" failed. No retries permitted until 2026-09-17 01:37:57.36376478 +0000 UTC m=+41.120532017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls") pod "image-registry-6c6587d777-q8lc4" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d") : secret "image-registry-tls" not found Sep 17 01:37:55.363984 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:55.363807 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert podName:5467c639-5928-4a5f-a109-18c20b43fcfa nodeName:}" failed. No retries permitted until 2026-09-17 01:37:57.363793508 +0000 UTC m=+41.120560744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-9r2gb" (UID: "5467c639-5928-4a5f-a109-18c20b43fcfa") : secret "networking-console-plugin-cert" not found Sep 17 01:37:57.379660 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:57.379454 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:37:57.380143 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:57.379695 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:37:57.380143 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:57.379726 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:37:57.380143 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:57.379621 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 01:37:57.380143 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:57.379780 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:37:57.380143 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:57.379812 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert podName:5467c639-5928-4a5f-a109-18c20b43fcfa nodeName:}" failed. No retries permitted until 2026-09-17 01:38:01.37979415 +0000 UTC m=+45.136561385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-9r2gb" (UID: "5467c639-5928-4a5f-a109-18c20b43fcfa") : secret "networking-console-plugin-cert" not found Sep 17 01:37:57.380143 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:57.379826 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:37:57.380143 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:57.379839 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls podName:ac42c232-24c7-4609-a169-07fe9bb4c2e6 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:01.379821247 +0000 UTC m=+45.136588475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls") pod "dns-default-pb4gq" (UID: "ac42c232-24c7-4609-a169-07fe9bb4c2e6") : secret "dns-default-metrics-tls" not found Sep 17 01:37:57.380143 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:57.379840 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c6587d777-q8lc4: secret "image-registry-tls" not found Sep 17 01:37:57.380143 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:37:57.379883 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls podName:d8ed7b30-477c-480e-888f-94a13fb9892d nodeName:}" failed. No retries permitted until 2026-09-17 01:38:01.379872561 +0000 UTC m=+45.136639784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls") pod "image-registry-6c6587d777-q8lc4" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d") : secret "image-registry-tls" not found Sep 17 01:37:58.037269 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:58.037224 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2djzb" event={"ID":"3ce012e5-f6f2-4313-9085-430520d3d06d","Type":"ContainerStarted","Data":"7791c4936eb7122232d2e3839c53a57e14207e16bface8256c4758d1f94f0920"} Sep 17 01:37:58.051196 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:37:58.051138 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2djzb" podStartSLOduration=2.119554791 podStartE2EDuration="5.051123011s" totalImagesPullingTime="2.93156822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:58.05052808 +0000 UTC m=+41.807295336" watchObservedRunningTime="2026-09-17 01:37:58.051123011 +0000 UTC m=+41.807890254" Sep 17 01:38:01.408131 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:01.408086 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:38:01.408490 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:01.408138 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:38:01.408490 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:01.408175 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:38:01.408490 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:01.408235 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:01.408490 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:01.408267 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 01:38:01.408490 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:01.408289 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:38:01.408490 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:01.408299 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls podName:ac42c232-24c7-4609-a169-07fe9bb4c2e6 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:09.408284115 +0000 UTC m=+53.165051338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls") pod "dns-default-pb4gq" (UID: "ac42c232-24c7-4609-a169-07fe9bb4c2e6") : secret "dns-default-metrics-tls" not found Sep 17 01:38:01.408490 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:01.408305 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c6587d777-q8lc4: secret "image-registry-tls" not found Sep 17 01:38:01.408490 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:01.408314 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert podName:5467c639-5928-4a5f-a109-18c20b43fcfa nodeName:}" failed. No retries permitted until 2026-09-17 01:38:09.408307845 +0000 UTC m=+53.165075067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-9r2gb" (UID: "5467c639-5928-4a5f-a109-18c20b43fcfa") : secret "networking-console-plugin-cert" not found Sep 17 01:38:01.408490 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:01.408341 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls podName:d8ed7b30-477c-480e-888f-94a13fb9892d nodeName:}" failed. No retries permitted until 2026-09-17 01:38:09.408328459 +0000 UTC m=+53.165095706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls") pod "image-registry-6c6587d777-q8lc4" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d") : secret "image-registry-tls" not found Sep 17 01:38:09.468638 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:09.468569 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:38:09.468638 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:09.468652 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:38:09.469180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:09.468695 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:38:09.469180 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:09.468732 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:09.469180 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:09.468781 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 01:38:09.469180 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:09.468794 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:38:09.469180 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:09.468812 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c6587d777-q8lc4: secret "image-registry-tls" not found Sep 17 01:38:09.469180 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:09.468801 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls podName:ac42c232-24c7-4609-a169-07fe9bb4c2e6 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:25.468785062 +0000 UTC m=+69.225552303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls") pod "dns-default-pb4gq" (UID: "ac42c232-24c7-4609-a169-07fe9bb4c2e6") : secret "dns-default-metrics-tls" not found Sep 17 01:38:09.469180 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:09.468870 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert podName:5467c639-5928-4a5f-a109-18c20b43fcfa nodeName:}" failed. No retries permitted until 2026-09-17 01:38:25.468857073 +0000 UTC m=+69.225624295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-9r2gb" (UID: "5467c639-5928-4a5f-a109-18c20b43fcfa") : secret "networking-console-plugin-cert" not found Sep 17 01:38:09.469180 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:09.468881 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls podName:d8ed7b30-477c-480e-888f-94a13fb9892d nodeName:}" failed. No retries permitted until 2026-09-17 01:38:25.468875426 +0000 UTC m=+69.225642647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls") pod "image-registry-6c6587d777-q8lc4" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d") : secret "image-registry-tls" not found Sep 17 01:38:19.014768 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:19.014735 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw7gk" Sep 17 01:38:21.454674 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.454620 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:38:21.457206 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.457186 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 01:38:21.468873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.468848 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31ad03fc-51f4-46b5-9a17-4d7f38513e84-original-pull-secret\") pod \"global-pull-secret-syncer-f62th\" (UID: \"31ad03fc-51f4-46b5-9a17-4d7f38513e84\") " pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:38:21.555485 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.555426 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:38:21.558284 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.558265 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 01:38:21.566407 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:21.566385 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 01:38:21.566502 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:21.566453 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs podName:26801bb3-4b53-4677-a222-d71afd225350 nodeName:}" failed. No retries permitted until 2026-09-17 01:39:25.566436417 +0000 UTC m=+129.323203639 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs") pod "network-metrics-daemon-bjxzr" (UID: "26801bb3-4b53-4677-a222-d71afd225350") : secret "metrics-daemon-secret" not found Sep 17 01:38:21.656473 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.656434 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4vj\" (UniqueName: \"kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj\") pod \"network-check-target-rs9g5\" (UID: \"beb5923e-36ae-4f98-8dca-590530716c12\") " pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:38:21.659210 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.659190 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 01:38:21.669566 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.669543 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 01:38:21.670615 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.670601 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f62th" Sep 17 01:38:21.680562 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.680186 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp4vj\" (UniqueName: \"kubernetes.io/projected/beb5923e-36ae-4f98-8dca-590530716c12-kube-api-access-lp4vj\") pod \"network-check-target-rs9g5\" (UID: \"beb5923e-36ae-4f98-8dca-590530716c12\") " pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:38:21.776483 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.776454 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hdsht\"" Sep 17 01:38:21.785343 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.785322 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:38:21.945312 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:21.945276 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f62th"] Sep 17 01:38:21.948603 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:38:21.948561 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ad03fc_51f4_46b5_9a17_4d7f38513e84.slice/crio-723cf2116aa6bb4923ebe15a276f4f7a7968f4f3af3bea25a6f1f4ff47cda7bb WatchSource:0}: Error finding container 723cf2116aa6bb4923ebe15a276f4f7a7968f4f3af3bea25a6f1f4ff47cda7bb: Status 404 returned error can't find the container with id 723cf2116aa6bb4923ebe15a276f4f7a7968f4f3af3bea25a6f1f4ff47cda7bb Sep 17 01:38:22.050382 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:22.050304 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rs9g5"] Sep 17 01:38:22.053482 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:38:22.053456 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb5923e_36ae_4f98_8dca_590530716c12.slice/crio-0b6cece35865a4f9cdce13f2d45549f1ce77203aacce4c25f022a51d987ca891 WatchSource:0}: Error finding container 0b6cece35865a4f9cdce13f2d45549f1ce77203aacce4c25f022a51d987ca891: Status 404 returned error can't find the container with id 0b6cece35865a4f9cdce13f2d45549f1ce77203aacce4c25f022a51d987ca891 Sep 17 01:38:22.095502 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:22.095464 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rs9g5" event={"ID":"beb5923e-36ae-4f98-8dca-590530716c12","Type":"ContainerStarted","Data":"0b6cece35865a4f9cdce13f2d45549f1ce77203aacce4c25f022a51d987ca891"} Sep 17 01:38:22.096614 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:22.096572 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f62th" event={"ID":"31ad03fc-51f4-46b5-9a17-4d7f38513e84","Type":"ContainerStarted","Data":"723cf2116aa6bb4923ebe15a276f4f7a7968f4f3af3bea25a6f1f4ff47cda7bb"} Sep 17 01:38:25.486640 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:25.486591 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:38:25.487241 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:25.486690 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:38:25.487241 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:25.486723 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:38:25.487241 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:25.486739 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 01:38:25.487241 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:25.486825 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert podName:5467c639-5928-4a5f-a109-18c20b43fcfa nodeName:}" failed. No retries permitted until 2026-09-17 01:38:57.486806136 +0000 UTC m=+101.243573374 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-9r2gb" (UID: "5467c639-5928-4a5f-a109-18c20b43fcfa") : secret "networking-console-plugin-cert" not found Sep 17 01:38:25.487241 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:25.486836 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:25.487241 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:25.486840 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:38:25.487241 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:25.486858 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c6587d777-q8lc4: secret "image-registry-tls" not found Sep 17 01:38:25.487241 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:25.486900 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls podName:d8ed7b30-477c-480e-888f-94a13fb9892d nodeName:}" failed. No retries permitted until 2026-09-17 01:38:57.486883356 +0000 UTC m=+101.243650603 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls") pod "image-registry-6c6587d777-q8lc4" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d") : secret "image-registry-tls" not found Sep 17 01:38:25.487241 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:25.486915 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls podName:ac42c232-24c7-4609-a169-07fe9bb4c2e6 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:57.486906488 +0000 UTC m=+101.243673711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls") pod "dns-default-pb4gq" (UID: "ac42c232-24c7-4609-a169-07fe9bb4c2e6") : secret "dns-default-metrics-tls" not found Sep 17 01:38:27.110739 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:27.110709 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rs9g5" event={"ID":"beb5923e-36ae-4f98-8dca-590530716c12","Type":"ContainerStarted","Data":"a2807df6400eb452e9adbf259ec55b0e91d51f849b2f70fda951d6e44bd7795a"} Sep 17 01:38:27.111167 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:27.110772 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:38:27.112125 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:27.112102 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f62th" event={"ID":"31ad03fc-51f4-46b5-9a17-4d7f38513e84","Type":"ContainerStarted","Data":"6ef95a677f9301ab90186c2c927ce8a3123bbcfa167ac2e8ee2315882533dc67"} Sep 17 01:38:27.126122 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:27.126074 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rs9g5" podStartSLOduration=65.528323182 podStartE2EDuration="1m10.126060185s" totalImagesPullingTime="4.597737003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:38:27.125608772 +0000 UTC m=+70.882376017" watchObservedRunningTime="2026-09-17 01:38:27.126060185 +0000 UTC m=+70.882827426" Sep 17 01:38:27.139209 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:27.139173 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f62th" podStartSLOduration=65.431656866 podStartE2EDuration="1m10.139161078s" totalImagesPullingTime="4.707504212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:38:27.13884583 +0000 UTC m=+70.895613107" watchObservedRunningTime="2026-09-17 01:38:27.139161078 +0000 UTC m=+70.895928334" Sep 17 01:38:57.521537 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:57.521496 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:38:57.522019 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:57.521546 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:38:57.522019 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:57.521612 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:38:57.522019 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:57.521651 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:57.522019 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:57.521674 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:38:57.522019 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:57.521691 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c6587d777-q8lc4: secret "image-registry-tls" not found Sep 17 01:38:57.522019 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:57.521723 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls podName:ac42c232-24c7-4609-a169-07fe9bb4c2e6 nodeName:}" failed. No retries permitted until 2026-09-17 01:40:01.521707576 +0000 UTC m=+165.278474815 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls") pod "dns-default-pb4gq" (UID: "ac42c232-24c7-4609-a169-07fe9bb4c2e6") : secret "dns-default-metrics-tls" not found Sep 17 01:38:57.522019 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:57.521728 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 01:38:57.522019 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:57.521742 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls podName:d8ed7b30-477c-480e-888f-94a13fb9892d nodeName:}" failed. No retries permitted until 2026-09-17 01:40:01.521730091 +0000 UTC m=+165.278497314 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls") pod "image-registry-6c6587d777-q8lc4" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d") : secret "image-registry-tls" not found Sep 17 01:38:57.522019 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:38:57.521772 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert podName:5467c639-5928-4a5f-a109-18c20b43fcfa nodeName:}" failed. No retries permitted until 2026-09-17 01:40:01.521758375 +0000 UTC m=+165.278525597 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-9r2gb" (UID: "5467c639-5928-4a5f-a109-18c20b43fcfa") : secret "networking-console-plugin-cert" not found Sep 17 01:38:58.117398 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:38:58.117368 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rs9g5" Sep 17 01:39:25.614674 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:39:25.614627 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:39:25.615143 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:39:25.614775 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 01:39:25.615143 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:39:25.614851 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs podName:26801bb3-4b53-4677-a222-d71afd225350 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:27.614835122 +0000 UTC m=+251.371602343 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs") pod "network-metrics-daemon-bjxzr" (UID: "26801bb3-4b53-4677-a222-d71afd225350") : secret "metrics-daemon-secret" not found Sep 17 01:39:56.630884 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:39:56.630836 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" podUID="5467c639-5928-4a5f-a109-18c20b43fcfa" Sep 17 01:39:56.656378 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:39:56.656345 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" podUID="d8ed7b30-477c-480e-888f-94a13fb9892d" Sep 17 01:39:56.694507 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:39:56.694465 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pb4gq" podUID="ac42c232-24c7-4609-a169-07fe9bb4c2e6" Sep 17 01:39:56.776134 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:39:56.776086 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:39:57.325995 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:39:57.325905 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pb4gq" Sep 17 01:39:57.325995 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:39:57.325905 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:40:01.568636 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:40:01.568556 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:40:01.568636 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:40:01.568641 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:40:01.569119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:40:01.568681 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:40:01.569119 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:40:01.568707 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:40:01.569119 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:40:01.568726 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c6587d777-q8lc4: secret "image-registry-tls" not found Sep 17 01:40:01.569119 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:40:01.568758 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:40:01.569119 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:40:01.568783 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls podName:d8ed7b30-477c-480e-888f-94a13fb9892d nodeName:}" failed. No retries permitted until 2026-09-17 01:42:03.56876572 +0000 UTC m=+287.325532942 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls") pod "image-registry-6c6587d777-q8lc4" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d") : secret "image-registry-tls" not found Sep 17 01:40:01.569119 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:40:01.568784 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 01:40:01.569119 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:40:01.568803 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls podName:ac42c232-24c7-4609-a169-07fe9bb4c2e6 nodeName:}" failed. No retries permitted until 2026-09-17 01:42:03.568792729 +0000 UTC m=+287.325559951 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls") pod "dns-default-pb4gq" (UID: "ac42c232-24c7-4609-a169-07fe9bb4c2e6") : secret "dns-default-metrics-tls" not found Sep 17 01:40:01.569119 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:40:01.568841 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert podName:5467c639-5928-4a5f-a109-18c20b43fcfa nodeName:}" failed. No retries permitted until 2026-09-17 01:42:03.568827575 +0000 UTC m=+287.325594797 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-9r2gb" (UID: "5467c639-5928-4a5f-a109-18c20b43fcfa") : secret "networking-console-plugin-cert" not found Sep 17 01:40:10.761576 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:40:10.761486 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:40:10.761576 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:40:10.761493 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:41:22.792224 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.792173 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs"] Sep 17 01:41:22.795223 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.795191 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q"] Sep 17 01:41:22.795332 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.795292 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:22.797522 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.797503 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 17 01:41:22.797981 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.797959 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-dcgdm"] Sep 17 01:41:22.798076 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.798020 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q" Sep 17 01:41:22.799114 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.799092 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 01:41:22.799257 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.799158 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9cxjc\"" Sep 17 01:41:22.799257 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.799179 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 01:41:22.799257 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.799231 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 17 01:41:22.800171 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.800152 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:41:22.800237 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.800192 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-brpm8\"" Sep 17 01:41:22.800237 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.800199 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 17 01:41:22.800509 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.800489 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xz5g9"] Sep 17 01:41:22.800638 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.800626 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:22.804404 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.803236 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 17 01:41:22.804404 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.803835 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 17 01:41:22.807027 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.806947 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-56rtn\"" Sep 17 01:41:22.807027 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.807014 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 01:41:22.808021 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.807995 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 01:41:22.808298 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.808278 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:22.809462 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.809441 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q"] Sep 17 01:41:22.810613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.810524 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs"] Sep 17 01:41:22.811708 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.811250 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-6b4gp\"" Sep 17 01:41:22.811708 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.811327 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 17 01:41:22.811708 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.811480 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 17 01:41:22.811708 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.811537 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-dcgdm"] Sep 17 01:41:22.812366 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.812349 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 17 01:41:22.812455 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.812347 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:41:22.812678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.812662 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 17 01:41:22.815147 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.815128 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xz5g9"] Sep 17 01:41:22.816787 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.816768 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 17 01:41:22.887009 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.886974 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf"] Sep 17 01:41:22.890115 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.890085 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:22.890850 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.890826 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c876f99bb-r6ww6"] Sep 17 01:41:22.892227 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.892209 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 17 01:41:22.893345 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.893324 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:41:22.893514 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.893499 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:22.893557 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.893529 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wxzpz\"" Sep 17 01:41:22.893761 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.893742 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 17 01:41:22.894998 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.894983 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 17 01:41:22.897059 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.897039 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 17 01:41:22.897168 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.897093 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 17 01:41:22.897232 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.897215 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 17 01:41:22.897284 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.897128 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 01:41:22.897284 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.897266 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 17 01:41:22.897495 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.897478 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 01:41:22.897566 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.897478 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fbs9x\"" Sep 17 01:41:22.901694 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.901673 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf"] Sep 17 01:41:22.909692 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.909671 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c876f99bb-r6ww6"] Sep 17 01:41:22.959526 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959485 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr29p\" (UniqueName: \"kubernetes.io/projected/0223775e-2dd4-4ab0-b979-49b697a046a4-kube-api-access-qr29p\") pod \"volume-data-source-validator-86695b7cc7-5p94q\" (UID: \"0223775e-2dd4-4ab0-b979-49b697a046a4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q" Sep 17 01:41:22.959526 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959527 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42508a-cdab-4526-8b1a-de7088e24b00-service-ca-bundle\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:22.959802 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959557 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d084eabd-5582-43e0-84bf-d5a860eb8092-serving-cert\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:22.959802 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959600 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a42508a-cdab-4526-8b1a-de7088e24b00-snapshots\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:22.959802 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959653 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdnn\" (UniqueName: \"kubernetes.io/projected/2a42508a-cdab-4526-8b1a-de7088e24b00-kube-api-access-vgdnn\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:22.959802 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959681 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:22.959802 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959705 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a42508a-cdab-4526-8b1a-de7088e24b00-serving-cert\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:22.959802 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959729 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42508a-cdab-4526-8b1a-de7088e24b00-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:22.959802 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959788 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtsjk\" (UniqueName: \"kubernetes.io/projected/d084eabd-5582-43e0-84bf-d5a860eb8092-kube-api-access-vtsjk\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:22.960039 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959842 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lgkn\" (UniqueName: \"kubernetes.io/projected/d637b01d-51f5-4846-a45e-6968f5411206-kube-api-access-4lgkn\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:22.960039 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959869 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a42508a-cdab-4526-8b1a-de7088e24b00-tmp\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:22.960039 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959890 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d084eabd-5582-43e0-84bf-d5a860eb8092-trusted-ca\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:22.960039 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959908 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d637b01d-51f5-4846-a45e-6968f5411206-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:22.960039 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:22.959936 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d084eabd-5582-43e0-84bf-d5a860eb8092-config\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:23.060674 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060559 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdnn\" (UniqueName: \"kubernetes.io/projected/2a42508a-cdab-4526-8b1a-de7088e24b00-kube-api-access-vgdnn\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.060674 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060632 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ca692a6-2629-40fb-a8fc-26571248db38-config\") pod \"service-ca-operator-cf7c776b7-w9lgf\" (UID: \"3ca692a6-2629-40fb-a8fc-26571248db38\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:23.060674 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060656 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-default-certificate\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.060674 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060675 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-stats-auth\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060694 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060712 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a42508a-cdab-4526-8b1a-de7088e24b00-serving-cert\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060733 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42508a-cdab-4526-8b1a-de7088e24b00-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060767 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtsjk\" (UniqueName: \"kubernetes.io/projected/d084eabd-5582-43e0-84bf-d5a860eb8092-kube-api-access-vtsjk\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060796 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d87v\" (UniqueName: \"kubernetes.io/projected/640963f2-fa1e-44ca-9aa4-e508696d9bf7-kube-api-access-7d87v\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:23.060809 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060831 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lgkn\" (UniqueName: \"kubernetes.io/projected/d637b01d-51f5-4846-a45e-6968f5411206-kube-api-access-4lgkn\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:23.060893 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls podName:d637b01d-51f5-4846-a45e-6968f5411206 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:23.560873137 +0000 UTC m=+247.317640370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mp6xs" (UID: "d637b01d-51f5-4846-a45e-6968f5411206") : secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060928 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a42508a-cdab-4526-8b1a-de7088e24b00-tmp\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060959 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692z8\" (UniqueName: \"kubernetes.io/projected/3ca692a6-2629-40fb-a8fc-26571248db38-kube-api-access-692z8\") pod \"service-ca-operator-cf7c776b7-w9lgf\" (UID: \"3ca692a6-2629-40fb-a8fc-26571248db38\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:23.060990 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.060991 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d084eabd-5582-43e0-84bf-d5a860eb8092-trusted-ca\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:23.061513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061017 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d637b01d-51f5-4846-a45e-6968f5411206-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:23.061513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061055 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d084eabd-5582-43e0-84bf-d5a860eb8092-config\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:23.061513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061193 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.061513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061275 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a42508a-cdab-4526-8b1a-de7088e24b00-tmp\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.061513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061292 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.061513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061336 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr29p\" (UniqueName: \"kubernetes.io/projected/0223775e-2dd4-4ab0-b979-49b697a046a4-kube-api-access-qr29p\") pod \"volume-data-source-validator-86695b7cc7-5p94q\" (UID: \"0223775e-2dd4-4ab0-b979-49b697a046a4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q" Sep 17 01:41:23.061513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061394 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42508a-cdab-4526-8b1a-de7088e24b00-service-ca-bundle\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.061513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061428 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca692a6-2629-40fb-a8fc-26571248db38-serving-cert\") pod \"service-ca-operator-cf7c776b7-w9lgf\" (UID: \"3ca692a6-2629-40fb-a8fc-26571248db38\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:23.061513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061470 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d084eabd-5582-43e0-84bf-d5a860eb8092-serving-cert\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:23.061513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061500 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a42508a-cdab-4526-8b1a-de7088e24b00-snapshots\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.062081 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061904 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d084eabd-5582-43e0-84bf-d5a860eb8092-config\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:23.062081 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.061981 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d637b01d-51f5-4846-a45e-6968f5411206-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:23.062081 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.062074 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42508a-cdab-4526-8b1a-de7088e24b00-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.062303 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.062281 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a42508a-cdab-4526-8b1a-de7088e24b00-snapshots\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.062350 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.062286 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a42508a-cdab-4526-8b1a-de7088e24b00-service-ca-bundle\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.062350 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.062317 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d084eabd-5582-43e0-84bf-d5a860eb8092-trusted-ca\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:23.063500 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.063483 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a42508a-cdab-4526-8b1a-de7088e24b00-serving-cert\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.064132 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.064109 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d084eabd-5582-43e0-84bf-d5a860eb8092-serving-cert\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:23.068951 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.068925 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtsjk\" (UniqueName: \"kubernetes.io/projected/d084eabd-5582-43e0-84bf-d5a860eb8092-kube-api-access-vtsjk\") pod \"console-operator-696884fb9c-xz5g9\" (UID: \"d084eabd-5582-43e0-84bf-d5a860eb8092\") " pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:23.069090 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.069069 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lgkn\" (UniqueName: \"kubernetes.io/projected/d637b01d-51f5-4846-a45e-6968f5411206-kube-api-access-4lgkn\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:23.069174 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.069152 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdnn\" (UniqueName: \"kubernetes.io/projected/2a42508a-cdab-4526-8b1a-de7088e24b00-kube-api-access-vgdnn\") pod \"insights-operator-5844b856db-dcgdm\" (UID: \"2a42508a-cdab-4526-8b1a-de7088e24b00\") " pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.069433 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.069415 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr29p\" (UniqueName: \"kubernetes.io/projected/0223775e-2dd4-4ab0-b979-49b697a046a4-kube-api-access-qr29p\") pod \"volume-data-source-validator-86695b7cc7-5p94q\" (UID: \"0223775e-2dd4-4ab0-b979-49b697a046a4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q" Sep 17 01:41:23.116470 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.116431 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q" Sep 17 01:41:23.123564 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.123547 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-dcgdm" Sep 17 01:41:23.128471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.128449 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:23.162836 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.162705 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.162836 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.162790 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.162836 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.162818 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca692a6-2629-40fb-a8fc-26571248db38-serving-cert\") pod \"service-ca-operator-cf7c776b7-w9lgf\" (UID: \"3ca692a6-2629-40fb-a8fc-26571248db38\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:23.163150 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.162862 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ca692a6-2629-40fb-a8fc-26571248db38-config\") pod \"service-ca-operator-cf7c776b7-w9lgf\" (UID: \"3ca692a6-2629-40fb-a8fc-26571248db38\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:23.163150 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:23.162891 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:23.662870528 +0000 UTC m=+247.419637774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : configmap references non-existent config key: service-ca.crt Sep 17 01:41:23.163150 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.162929 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-default-certificate\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.163150 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.162955 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-stats-auth\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.163150 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.163006 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d87v\" (UniqueName: \"kubernetes.io/projected/640963f2-fa1e-44ca-9aa4-e508696d9bf7-kube-api-access-7d87v\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.163150 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.163042 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692z8\" (UniqueName: \"kubernetes.io/projected/3ca692a6-2629-40fb-a8fc-26571248db38-kube-api-access-692z8\") pod \"service-ca-operator-cf7c776b7-w9lgf\" (UID: \"3ca692a6-2629-40fb-a8fc-26571248db38\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:23.163902 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.163457 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ca692a6-2629-40fb-a8fc-26571248db38-config\") pod \"service-ca-operator-cf7c776b7-w9lgf\" (UID: \"3ca692a6-2629-40fb-a8fc-26571248db38\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:23.163902 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:23.163848 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:41:23.164072 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:23.163910 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:23.663891691 +0000 UTC m=+247.420658932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : secret "router-metrics-certs-default" not found Sep 17 01:41:23.166692 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.166640 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca692a6-2629-40fb-a8fc-26571248db38-serving-cert\") pod \"service-ca-operator-cf7c776b7-w9lgf\" (UID: \"3ca692a6-2629-40fb-a8fc-26571248db38\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:23.166868 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.166848 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-stats-auth\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.167642 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.167621 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-default-certificate\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.180321 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.179022 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-692z8\" (UniqueName: \"kubernetes.io/projected/3ca692a6-2629-40fb-a8fc-26571248db38-kube-api-access-692z8\") pod \"service-ca-operator-cf7c776b7-w9lgf\" (UID: \"3ca692a6-2629-40fb-a8fc-26571248db38\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:23.185831 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.185802 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d87v\" (UniqueName: \"kubernetes.io/projected/640963f2-fa1e-44ca-9aa4-e508696d9bf7-kube-api-access-7d87v\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.199906 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.199778 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" Sep 17 01:41:23.393439 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.393403 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q"] Sep 17 01:41:23.396877 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:41:23.396852 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0223775e_2dd4_4ab0_b979_49b697a046a4.slice/crio-df884455917db63964a91b6eb22beec84997657ad11a7d6d0be13f95dfa20286 WatchSource:0}: Error finding container df884455917db63964a91b6eb22beec84997657ad11a7d6d0be13f95dfa20286: Status 404 returned error can't find the container with id df884455917db63964a91b6eb22beec84997657ad11a7d6d0be13f95dfa20286 Sep 17 01:41:23.401558 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.401528 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-dcgdm"] Sep 17 01:41:23.403913 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:41:23.403882 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a42508a_cdab_4526_8b1a_de7088e24b00.slice/crio-d67e1240f48c473dd7bbb1da5be5f59035d14ece6cebce8d9bb084d986b5684d WatchSource:0}: Error finding container d67e1240f48c473dd7bbb1da5be5f59035d14ece6cebce8d9bb084d986b5684d: Status 404 returned error can't find the container with id d67e1240f48c473dd7bbb1da5be5f59035d14ece6cebce8d9bb084d986b5684d Sep 17 01:41:23.425887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.425850 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xz5g9"] Sep 17 01:41:23.429516 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:41:23.429490 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd084eabd_5582_43e0_84bf_d5a860eb8092.slice/crio-b7d9d347e76f0d787ebab7d46d290b28fe8e609757b5c7d7f8f9f6c96e681854 WatchSource:0}: Error finding container b7d9d347e76f0d787ebab7d46d290b28fe8e609757b5c7d7f8f9f6c96e681854: Status 404 returned error can't find the container with id b7d9d347e76f0d787ebab7d46d290b28fe8e609757b5c7d7f8f9f6c96e681854 Sep 17 01:41:23.485914 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.485884 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf"] Sep 17 01:41:23.489037 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:41:23.489017 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca692a6_2629_40fb_a8fc_26571248db38.slice/crio-580453b3a5f70e4a1d5d0e1d21b3586a2c3444efdf50af5925824902da1eb913 WatchSource:0}: Error finding container 580453b3a5f70e4a1d5d0e1d21b3586a2c3444efdf50af5925824902da1eb913: Status 404 returned error can't find the container with id 580453b3a5f70e4a1d5d0e1d21b3586a2c3444efdf50af5925824902da1eb913 Sep 17 01:41:23.530806 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.530766 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" event={"ID":"3ca692a6-2629-40fb-a8fc-26571248db38","Type":"ContainerStarted","Data":"580453b3a5f70e4a1d5d0e1d21b3586a2c3444efdf50af5925824902da1eb913"} Sep 17 01:41:23.531860 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.531825 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dcgdm" event={"ID":"2a42508a-cdab-4526-8b1a-de7088e24b00","Type":"ContainerStarted","Data":"d67e1240f48c473dd7bbb1da5be5f59035d14ece6cebce8d9bb084d986b5684d"} Sep 17 01:41:23.532957 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.532931 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" event={"ID":"d084eabd-5582-43e0-84bf-d5a860eb8092","Type":"ContainerStarted","Data":"b7d9d347e76f0d787ebab7d46d290b28fe8e609757b5c7d7f8f9f6c96e681854"} Sep 17 01:41:23.533895 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.533872 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q" event={"ID":"0223775e-2dd4-4ab0-b979-49b697a046a4","Type":"ContainerStarted","Data":"df884455917db63964a91b6eb22beec84997657ad11a7d6d0be13f95dfa20286"} Sep 17 01:41:23.565382 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.565343 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:23.565507 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:23.565493 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:23.565611 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:23.565598 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls podName:d637b01d-51f5-4846-a45e-6968f5411206 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:24.565558732 +0000 UTC m=+248.322325953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mp6xs" (UID: "d637b01d-51f5-4846-a45e-6968f5411206") : secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:23.666333 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.666243 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.666481 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:23.666353 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:23.666481 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:23.666387 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:41:23.666481 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:23.666453 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:24.666437897 +0000 UTC m=+248.423205119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : secret "router-metrics-certs-default" not found Sep 17 01:41:23.666481 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:23.666473 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:24.666460312 +0000 UTC m=+248.423227533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : configmap references non-existent config key: service-ca.crt Sep 17 01:41:24.575054 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:24.574948 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:24.575555 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:24.575130 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:24.575555 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:24.575225 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls podName:d637b01d-51f5-4846-a45e-6968f5411206 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:26.575197542 +0000 UTC m=+250.331964771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mp6xs" (UID: "d637b01d-51f5-4846-a45e-6968f5411206") : secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:24.676468 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:24.676425 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:24.676658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:24.676514 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:24.676809 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:24.676737 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:41:24.676809 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:24.676803 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:26.676783976 +0000 UTC m=+250.433551201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : secret "router-metrics-certs-default" not found Sep 17 01:41:24.677250 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:24.677228 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:26.677210189 +0000 UTC m=+250.433977416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : configmap references non-existent config key: service-ca.crt Sep 17 01:41:25.542259 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:25.542213 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q" event={"ID":"0223775e-2dd4-4ab0-b979-49b697a046a4","Type":"ContainerStarted","Data":"eb582aae85f32675736cbb0079804342ad1a5c1faa1cb0a42fe13aa56a4861ad"} Sep 17 01:41:25.556419 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:25.556363 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5p94q" podStartSLOduration=2.236953603 podStartE2EDuration="3.556340291s" totalImagesPullingTime="1.319386688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:25.555857567 +0000 UTC m=+249.312624844" watchObservedRunningTime="2026-09-17 01:41:25.556340291 +0000 UTC m=+249.313107538" Sep 17 01:41:26.596604 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:26.596016 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:26.596604 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:26.596205 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:26.596604 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:26.596267 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls podName:d637b01d-51f5-4846-a45e-6968f5411206 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:30.596248561 +0000 UTC m=+254.353015797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mp6xs" (UID: "d637b01d-51f5-4846-a45e-6968f5411206") : secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:26.696369 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:26.696333 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:26.696517 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:26.696390 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:26.696555 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:26.696521 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:41:26.696555 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:26.696533 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:30.696510353 +0000 UTC m=+254.453277591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : configmap references non-existent config key: service-ca.crt Sep 17 01:41:26.696673 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:26.696565 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:30.696553541 +0000 UTC m=+254.453320781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : secret "router-metrics-certs-default" not found Sep 17 01:41:27.551648 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.551602 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" event={"ID":"3ca692a6-2629-40fb-a8fc-26571248db38","Type":"ContainerStarted","Data":"53613ba4f8249b0913221f5e1431e9d245faa1217604a1e40b693ec15e608378"} Sep 17 01:41:27.553679 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.553627 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dcgdm" event={"ID":"2a42508a-cdab-4526-8b1a-de7088e24b00","Type":"ContainerStarted","Data":"657e88db405c5e932f13fc5a5a13b5b194374191c688ef78949386add4d24cd8"} Sep 17 01:41:27.556797 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.556772 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/0.log" Sep 17 01:41:27.556949 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.556838 2586 generic.go:356] "Generic (PLEG): container finished" podID="d084eabd-5582-43e0-84bf-d5a860eb8092" containerID="76eeb379ced5e92299632ff41e3d10f4a84e834fe216eab8b8ad04567f02444f" exitCode=255 Sep 17 01:41:27.556949 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.556912 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" event={"ID":"d084eabd-5582-43e0-84bf-d5a860eb8092","Type":"ContainerDied","Data":"76eeb379ced5e92299632ff41e3d10f4a84e834fe216eab8b8ad04567f02444f"} Sep 17 01:41:27.557253 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.557226 2586 scope.go:117] "RemoveContainer" containerID="76eeb379ced5e92299632ff41e3d10f4a84e834fe216eab8b8ad04567f02444f" Sep 17 01:41:27.570117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.570056 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" podStartSLOduration=2.554978133 podStartE2EDuration="5.570031938s" totalImagesPullingTime="3.015053805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:27.5685044 +0000 UTC m=+251.325271671" watchObservedRunningTime="2026-09-17 01:41:27.570031938 +0000 UTC m=+251.326799179" Sep 17 01:41:27.581607 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.580701 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kz697"] Sep 17 01:41:27.587849 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.586507 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-dcgdm" podStartSLOduration=2.491170424 podStartE2EDuration="5.586487314s" totalImagesPullingTime="3.09531689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:27.58581114 +0000 UTC m=+251.342578407" watchObservedRunningTime="2026-09-17 01:41:27.586487314 +0000 UTC m=+251.343254558" Sep 17 01:41:27.588089 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.588062 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.590564 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.590540 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 01:41:27.590697 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.590540 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 01:41:27.590775 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.590720 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l4q7v\"" Sep 17 01:41:27.600893 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.600851 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kz697"] Sep 17 01:41:27.605049 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.604857 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-crio-socket\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.605195 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.605081 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjw5\" (UniqueName: \"kubernetes.io/projected/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-api-access-ztjw5\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.605267 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.605231 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.605382 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.605298 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.605463 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.605430 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-data-volume\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.705757 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.705726 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:41:27.705757 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.705761 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjw5\" (UniqueName: \"kubernetes.io/projected/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-api-access-ztjw5\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.705971 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.705803 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.705971 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.705842 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.705971 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:27.705899 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 01:41:27.705971 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.705904 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-data-volume\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.705971 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.705953 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-crio-socket\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.705971 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:27.705960 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:41:27.706254 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:27.705974 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:27.706254 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:27.705980 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs podName:26801bb3-4b53-4677-a222-d71afd225350 nodeName:}" failed. No retries permitted until 2026-09-17 01:43:29.705957826 +0000 UTC m=+373.462725052 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs") pod "network-metrics-daemon-bjxzr" (UID: "26801bb3-4b53-4677-a222-d71afd225350") : secret "metrics-daemon-secret" not found Sep 17 01:41:27.706254 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:27.706017 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:28.206001355 +0000 UTC m=+251.962768595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:27.706254 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.706017 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-crio-socket\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.706254 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:27.706044 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:28.206034006 +0000 UTC m=+251.962801229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : configmap "kube-rbac-proxy" not found Sep 17 01:41:27.706436 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.706268 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-data-volume\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:27.713682 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:27.713661 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjw5\" (UniqueName: \"kubernetes.io/projected/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-api-access-ztjw5\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:28.209988 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:28.209926 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:28.210201 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:28.209999 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:28.210201 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:28.210060 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:41:28.210201 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:28.210100 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:28.210201 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:28.210133 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:29.210117263 +0000 UTC m=+252.966884485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : configmap "kube-rbac-proxy" not found Sep 17 01:41:28.210201 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:28.210151 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:29.210139483 +0000 UTC m=+252.966906705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:28.561833 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:28.561754 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/1.log" Sep 17 01:41:28.562169 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:28.562155 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/0.log" Sep 17 01:41:28.562224 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:28.562194 2586 generic.go:356] "Generic (PLEG): container finished" podID="d084eabd-5582-43e0-84bf-d5a860eb8092" containerID="334746549f6cc37598363ff58e0f0509b6143e528467fd6132d94a69168b16ea" exitCode=255 Sep 17 01:41:28.562274 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:28.562235 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" event={"ID":"d084eabd-5582-43e0-84bf-d5a860eb8092","Type":"ContainerDied","Data":"334746549f6cc37598363ff58e0f0509b6143e528467fd6132d94a69168b16ea"} Sep 17 01:41:28.562329 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:28.562316 2586 scope.go:117] "RemoveContainer" containerID="76eeb379ced5e92299632ff41e3d10f4a84e834fe216eab8b8ad04567f02444f" Sep 17 01:41:28.562519 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:28.562498 2586 scope.go:117] "RemoveContainer" containerID="334746549f6cc37598363ff58e0f0509b6143e528467fd6132d94a69168b16ea" Sep 17 01:41:28.562754 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:28.562729 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xz5g9_openshift-console-operator(d084eabd-5582-43e0-84bf-d5a860eb8092)\"" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" podUID="d084eabd-5582-43e0-84bf-d5a860eb8092" Sep 17 01:41:29.218747 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:29.218698 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:29.218747 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:29.218755 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:29.219310 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:29.218840 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:41:29.219310 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:29.218873 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:29.219310 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:29.218919 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:31.218904099 +0000 UTC m=+254.975671322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:29.219310 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:29.218931 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:31.218925223 +0000 UTC m=+254.975692445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : configmap "kube-rbac-proxy" not found Sep 17 01:41:29.566809 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:29.566723 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/1.log" Sep 17 01:41:29.567074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:29.567057 2586 scope.go:117] "RemoveContainer" containerID="334746549f6cc37598363ff58e0f0509b6143e528467fd6132d94a69168b16ea" Sep 17 01:41:29.567230 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:29.567211 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xz5g9_openshift-console-operator(d084eabd-5582-43e0-84bf-d5a860eb8092)\"" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" podUID="d084eabd-5582-43e0-84bf-d5a860eb8092" Sep 17 01:41:30.358061 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:30.358030 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k5mbc_4c2f9cad-e927-484e-9599-a9a84ade862e/dns-node-resolver/0.log" Sep 17 01:41:30.632437 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:30.632353 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:30.632602 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:30.632468 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:30.632602 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:30.632523 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls podName:d637b01d-51f5-4846-a45e-6968f5411206 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:38.632509584 +0000 UTC m=+262.389276806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mp6xs" (UID: "d637b01d-51f5-4846-a45e-6968f5411206") : secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:30.733150 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:30.733108 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:30.733282 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:30.733183 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:30.733318 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:30.733290 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:41:30.733353 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:30.733293 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:38.733273935 +0000 UTC m=+262.490041160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : configmap references non-existent config key: service-ca.crt Sep 17 01:41:30.733353 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:30.733351 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:38.733333191 +0000 UTC m=+262.490100414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : secret "router-metrics-certs-default" not found Sep 17 01:41:31.237086 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:31.237027 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:31.237277 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:31.237106 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:31.237277 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:31.237162 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:41:31.237277 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:31.237233 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:35.237215249 +0000 UTC m=+258.993982492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : configmap "kube-rbac-proxy" not found Sep 17 01:41:31.237277 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:31.237257 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:31.237417 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:31.237309 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:35.237296049 +0000 UTC m=+258.994063293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:31.360995 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:31.360966 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rf6rr_d9b4c988-c58d-4e30-94e9-a892d64598b4/node-ca/0.log" Sep 17 01:41:31.958413 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:31.958383 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2djzb_3ce012e5-f6f2-4313-9085-430520d3d06d/serve-healthcheck-canary/0.log" Sep 17 01:41:33.129680 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:33.129652 2586 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:33.130045 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:33.129707 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:33.130045 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:33.130008 2586 scope.go:117] "RemoveContainer" containerID="334746549f6cc37598363ff58e0f0509b6143e528467fd6132d94a69168b16ea" Sep 17 01:41:33.130170 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:33.130151 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xz5g9_openshift-console-operator(d084eabd-5582-43e0-84bf-d5a860eb8092)\"" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" podUID="d084eabd-5582-43e0-84bf-d5a860eb8092" Sep 17 01:41:35.271065 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:35.271022 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:35.271459 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:35.271088 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:35.271459 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:35.271162 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:41:35.271459 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:35.271239 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:43.271220807 +0000 UTC m=+267.027988029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : configmap "kube-rbac-proxy" not found Sep 17 01:41:35.271459 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:35.271255 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:35.271459 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:35.271364 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:43.271350325 +0000 UTC m=+267.028117547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:38.701856 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:38.701760 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:38.702224 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:38.701883 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:38.702224 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:38.701940 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls podName:d637b01d-51f5-4846-a45e-6968f5411206 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:54.701924996 +0000 UTC m=+278.458692231 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mp6xs" (UID: "d637b01d-51f5-4846-a45e-6968f5411206") : secret "cluster-monitoring-operator-tls" not found Sep 17 01:41:38.802360 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:38.802328 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:38.802536 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:38.802484 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:38.802536 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:38.802487 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:41:38.802671 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:38.802611 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:54.802562075 +0000 UTC m=+278.559329312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : secret "router-metrics-certs-default" not found Sep 17 01:41:38.802671 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:38.802637 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle podName:640963f2-fa1e-44ca-9aa4-e508696d9bf7 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:54.802626585 +0000 UTC m=+278.559393810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle") pod "router-default-5c876f99bb-r6ww6" (UID: "640963f2-fa1e-44ca-9aa4-e508696d9bf7") : configmap references non-existent config key: service-ca.crt Sep 17 01:41:43.336727 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:43.336683 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:43.336727 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:43.336733 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:43.337164 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:43.336912 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:43.337164 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:43.336972 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls podName:fa85611e-a030-4660-aeb8-5f11fbbdcfc0 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:59.336957437 +0000 UTC m=+283.093724659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz697" (UID: "fa85611e-a030-4660-aeb8-5f11fbbdcfc0") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:43.337237 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:43.337191 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:47.762365 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:47.762334 2586 scope.go:117] "RemoveContainer" containerID="334746549f6cc37598363ff58e0f0509b6143e528467fd6132d94a69168b16ea" Sep 17 01:41:48.623656 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:48.623627 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:41:48.624061 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:48.624045 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/1.log" Sep 17 01:41:48.624129 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:48.624081 2586 generic.go:356] "Generic (PLEG): container finished" podID="d084eabd-5582-43e0-84bf-d5a860eb8092" containerID="2599d224b492a4e85715acf4de62e39ba20b973e2042338661e93022522326d4" exitCode=255 Sep 17 01:41:48.624170 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:48.624129 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" event={"ID":"d084eabd-5582-43e0-84bf-d5a860eb8092","Type":"ContainerDied","Data":"2599d224b492a4e85715acf4de62e39ba20b973e2042338661e93022522326d4"} Sep 17 01:41:48.624207 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:48.624182 2586 scope.go:117] "RemoveContainer" containerID="334746549f6cc37598363ff58e0f0509b6143e528467fd6132d94a69168b16ea" Sep 17 01:41:48.624516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:48.624501 2586 scope.go:117] "RemoveContainer" containerID="2599d224b492a4e85715acf4de62e39ba20b973e2042338661e93022522326d4" Sep 17 01:41:48.624747 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:48.624722 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-xz5g9_openshift-console-operator(d084eabd-5582-43e0-84bf-d5a860eb8092)\"" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" podUID="d084eabd-5582-43e0-84bf-d5a860eb8092" Sep 17 01:41:49.628913 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:49.628884 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:41:53.129626 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:53.129575 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:53.130027 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:53.129649 2586 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:41:53.130027 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:53.129926 2586 scope.go:117] "RemoveContainer" containerID="2599d224b492a4e85715acf4de62e39ba20b973e2042338661e93022522326d4" Sep 17 01:41:53.130111 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:41:53.130091 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-xz5g9_openshift-console-operator(d084eabd-5582-43e0-84bf-d5a860eb8092)\"" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" podUID="d084eabd-5582-43e0-84bf-d5a860eb8092" Sep 17 01:41:54.729636 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:54.729563 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:54.732096 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:54.732066 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d637b01d-51f5-4846-a45e-6968f5411206-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mp6xs\" (UID: \"d637b01d-51f5-4846-a45e-6968f5411206\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:54.830876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:54.830833 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:54.831011 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:54.830912 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:54.831531 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:54.831504 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/640963f2-fa1e-44ca-9aa4-e508696d9bf7-service-ca-bundle\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:54.833310 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:54.833285 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640963f2-fa1e-44ca-9aa4-e508696d9bf7-metrics-certs\") pod \"router-default-5c876f99bb-r6ww6\" (UID: \"640963f2-fa1e-44ca-9aa4-e508696d9bf7\") " pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:54.907302 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:54.907277 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" Sep 17 01:41:55.006140 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:55.006063 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:55.173533 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:55.173509 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs"] Sep 17 01:41:55.176510 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:41:55.176487 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd637b01d_51f5_4846_a45e_6968f5411206.slice/crio-5c7af8776a6bd62fb3f7e455b98aa22b22e58b596d10ee1367c6d59cbf7866b5 WatchSource:0}: Error finding container 5c7af8776a6bd62fb3f7e455b98aa22b22e58b596d10ee1367c6d59cbf7866b5: Status 404 returned error can't find the container with id 5c7af8776a6bd62fb3f7e455b98aa22b22e58b596d10ee1367c6d59cbf7866b5 Sep 17 01:41:55.274700 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:55.274680 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c876f99bb-r6ww6"] Sep 17 01:41:55.276621 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:41:55.276595 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640963f2_fa1e_44ca_9aa4_e508696d9bf7.slice/crio-c23ac463b0f3c34383bd98ed696e3260f9eda8be73e4cf87fb3608dfce5f788a WatchSource:0}: Error finding container c23ac463b0f3c34383bd98ed696e3260f9eda8be73e4cf87fb3608dfce5f788a: Status 404 returned error can't find the container with id c23ac463b0f3c34383bd98ed696e3260f9eda8be73e4cf87fb3608dfce5f788a Sep 17 01:41:55.648941 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:55.648899 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c876f99bb-r6ww6" event={"ID":"640963f2-fa1e-44ca-9aa4-e508696d9bf7","Type":"ContainerStarted","Data":"8e2d4238483b8a9d89daa93913151869a0a0961fb59e0560271e933cfe151122"} Sep 17 01:41:55.649102 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:55.648948 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c876f99bb-r6ww6" event={"ID":"640963f2-fa1e-44ca-9aa4-e508696d9bf7","Type":"ContainerStarted","Data":"c23ac463b0f3c34383bd98ed696e3260f9eda8be73e4cf87fb3608dfce5f788a"} Sep 17 01:41:55.650079 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:55.650057 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" event={"ID":"d637b01d-51f5-4846-a45e-6968f5411206","Type":"ContainerStarted","Data":"5c7af8776a6bd62fb3f7e455b98aa22b22e58b596d10ee1367c6d59cbf7866b5"} Sep 17 01:41:55.667278 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:55.667221 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5c876f99bb-r6ww6" podStartSLOduration=33.667207187 podStartE2EDuration="33.667207187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:55.666356011 +0000 UTC m=+279.423123278" watchObservedRunningTime="2026-09-17 01:41:55.667207187 +0000 UTC m=+279.423974431" Sep 17 01:41:56.007173 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:56.007097 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:56.010066 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:56.010040 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:56.654618 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:56.654567 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:56.656079 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:56.656056 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c876f99bb-r6ww6" Sep 17 01:41:57.659517 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:57.659475 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" event={"ID":"d637b01d-51f5-4846-a45e-6968f5411206","Type":"ContainerStarted","Data":"6f6f506f128cb9c4d52cb3dd0f724409e1e608c7503bc501946d180fa5e6a8fe"} Sep 17 01:41:57.675224 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:57.675174 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mp6xs" podStartSLOduration=33.943856128 podStartE2EDuration="35.675161221s" totalImagesPullingTime="1.731305093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:57.673833399 +0000 UTC m=+281.430600669" watchObservedRunningTime="2026-09-17 01:41:57.675161221 +0000 UTC m=+281.431928464" Sep 17 01:41:59.371524 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:59.371487 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:59.373925 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:59.373895 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa85611e-a030-4660-aeb8-5f11fbbdcfc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz697\" (UID: \"fa85611e-a030-4660-aeb8-5f11fbbdcfc0\") " pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:59.412022 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:59.411999 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kz697" Sep 17 01:41:59.721369 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:41:59.721301 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kz697"] Sep 17 01:41:59.724362 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:41:59.724330 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa85611e_a030_4660_aeb8_5f11fbbdcfc0.slice/crio-6a0a8e9abd385bbe54dd91da42a6aa449c0626a9601a45cbf00f3e918b7e9491 WatchSource:0}: Error finding container 6a0a8e9abd385bbe54dd91da42a6aa449c0626a9601a45cbf00f3e918b7e9491: Status 404 returned error can't find the container with id 6a0a8e9abd385bbe54dd91da42a6aa449c0626a9601a45cbf00f3e918b7e9491 Sep 17 01:42:00.327376 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:42:00.327335 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" podUID="5467c639-5928-4a5f-a109-18c20b43fcfa" Sep 17 01:42:00.327376 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:42:00.327335 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pb4gq" podUID="ac42c232-24c7-4609-a169-07fe9bb4c2e6" Sep 17 01:42:00.670500 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:00.670470 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pb4gq" Sep 17 01:42:00.670910 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:00.670503 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:42:00.670910 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:00.670464 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kz697" event={"ID":"fa85611e-a030-4660-aeb8-5f11fbbdcfc0","Type":"ContainerStarted","Data":"77a7a4a6f4ecebfec0a4e7b02dde425f0bd0216c1dc1c3582369ac81975bb60a"} Sep 17 01:42:00.670910 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:00.670626 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kz697" event={"ID":"fa85611e-a030-4660-aeb8-5f11fbbdcfc0","Type":"ContainerStarted","Data":"6a0a8e9abd385bbe54dd91da42a6aa449c0626a9601a45cbf00f3e918b7e9491"} Sep 17 01:42:01.675996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:01.675957 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kz697" event={"ID":"fa85611e-a030-4660-aeb8-5f11fbbdcfc0","Type":"ContainerStarted","Data":"a939be3da62def77fd647e2e5ec516d878986137c5c69b2afc5221cefbf95773"} Sep 17 01:42:02.680800 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:02.680767 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kz697" event={"ID":"fa85611e-a030-4660-aeb8-5f11fbbdcfc0","Type":"ContainerStarted","Data":"ada2a1f3cd3c55060d99f70af33bc9a73ad55b5aa53f4643b76986523535b359"} Sep 17 01:42:02.697234 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:02.697191 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kz697" podStartSLOduration=33.403275375 podStartE2EDuration="35.697176837s" totalImagesPullingTime="2.293901462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:41:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:02.696619199 +0000 UTC m=+286.453386453" watchObservedRunningTime="2026-09-17 01:42:02.697176837 +0000 UTC m=+286.453944081" Sep 17 01:42:03.604122 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.604093 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:42:03.604122 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.604126 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:42:03.604373 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.604151 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:42:03.606602 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.606554 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c232-24c7-4609-a169-07fe9bb4c2e6-metrics-tls\") pod \"dns-default-pb4gq\" (UID: \"ac42c232-24c7-4609-a169-07fe9bb4c2e6\") " pod="openshift-dns/dns-default-pb4gq" Sep 17 01:42:03.606704 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.606658 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"image-registry-6c6587d777-q8lc4\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:42:03.606704 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.606668 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5467c639-5928-4a5f-a109-18c20b43fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9r2gb\" (UID: \"5467c639-5928-4a5f-a109-18c20b43fcfa\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:42:03.674135 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.674111 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gjn79\"" Sep 17 01:42:03.675006 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.674991 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-crsck\"" Sep 17 01:42:03.681866 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.681853 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pb4gq" Sep 17 01:42:03.682147 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.681962 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" Sep 17 01:42:03.762258 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.762186 2586 scope.go:117] "RemoveContainer" containerID="2599d224b492a4e85715acf4de62e39ba20b973e2042338661e93022522326d4" Sep 17 01:42:03.762487 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:42:03.762349 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-xz5g9_openshift-console-operator(d084eabd-5582-43e0-84bf-d5a860eb8092)\"" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" podUID="d084eabd-5582-43e0-84bf-d5a860eb8092" Sep 17 01:42:03.864550 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.864490 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7d5d6\"" Sep 17 01:42:03.873336 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.873318 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:42:03.959737 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.959712 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb"] Sep 17 01:42:03.962526 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:03.962492 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pb4gq"] Sep 17 01:42:04.144735 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:04.144661 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c6587d777-q8lc4"] Sep 17 01:42:04.147225 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:42:04.147201 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ed7b30_477c_480e_888f_94a13fb9892d.slice/crio-f3c6205337183368ad90fe6029b32264f7fda38597b4fcd14eef7acb7b3ea497 WatchSource:0}: Error finding container f3c6205337183368ad90fe6029b32264f7fda38597b4fcd14eef7acb7b3ea497: Status 404 returned error can't find the container with id f3c6205337183368ad90fe6029b32264f7fda38597b4fcd14eef7acb7b3ea497 Sep 17 01:42:04.689826 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:04.689788 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" event={"ID":"d8ed7b30-477c-480e-888f-94a13fb9892d","Type":"ContainerStarted","Data":"d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6"} Sep 17 01:42:04.690300 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:04.689853 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" event={"ID":"d8ed7b30-477c-480e-888f-94a13fb9892d","Type":"ContainerStarted","Data":"f3c6205337183368ad90fe6029b32264f7fda38597b4fcd14eef7acb7b3ea497"} Sep 17 01:42:04.690300 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:04.689910 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:42:04.691073 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:04.691037 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" event={"ID":"5467c639-5928-4a5f-a109-18c20b43fcfa","Type":"ContainerStarted","Data":"cf2da2832344f2a359c61fd672d038d99eb1aaaaf259ca8e6ecf4d216b258773"} Sep 17 01:42:04.692224 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:04.692200 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pb4gq" event={"ID":"ac42c232-24c7-4609-a169-07fe9bb4c2e6","Type":"ContainerStarted","Data":"b5a6c700a2156414bacc698861f64b7a3084bc84c0e36bb65f2033e60810ce8c"} Sep 17 01:42:04.708602 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:04.708536 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" podStartSLOduration=286.708524422 podStartE2EDuration="4m46.708524422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:04.70748783 +0000 UTC m=+288.464255119" watchObservedRunningTime="2026-09-17 01:42:04.708524422 +0000 UTC m=+288.465291665" Sep 17 01:42:05.698417 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.698344 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" event={"ID":"5467c639-5928-4a5f-a109-18c20b43fcfa","Type":"ContainerStarted","Data":"bfdbb7434770b638a5d5fcaaad7616f0d1e97d991495c2f1284802af069da9d3"} Sep 17 01:42:05.724408 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.724334 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9r2gb" podStartSLOduration=275.49609424 podStartE2EDuration="4m36.724314836s" totalImagesPullingTime="1.228220596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:05.722920235 +0000 UTC m=+289.479687503" watchObservedRunningTime="2026-09-17 01:42:05.724314836 +0000 UTC m=+289.481082084" Sep 17 01:42:05.769867 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.769837 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75"] Sep 17 01:42:05.772935 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.772914 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:05.775901 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.775884 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 17 01:42:05.776066 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.776049 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 01:42:05.776408 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.776384 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 17 01:42:05.776408 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.776392 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8x6sx\"" Sep 17 01:42:05.781814 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.781794 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4ckfg"] Sep 17 01:42:05.785887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.785868 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77"] Sep 17 01:42:05.786020 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.786002 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:05.788952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.788526 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 01:42:05.788952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.788615 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9bxmv\"" Sep 17 01:42:05.788952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.788802 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 01:42:05.788952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.788843 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75"] Sep 17 01:42:05.788952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.788843 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 01:42:05.788952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.788915 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:05.791440 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.791372 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 17 01:42:05.791440 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.791396 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 17 01:42:05.791647 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.791442 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 17 01:42:05.791647 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.791488 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-c6bm5\"" Sep 17 01:42:05.797472 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.797452 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77"] Sep 17 01:42:05.922166 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922138 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-wtmp\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:05.922282 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922175 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-tls\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:05.922282 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922196 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0950289d-133e-441f-9ea8-ccc434f1c101-sys\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:05.922282 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922229 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-textfile\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:05.922282 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922259 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/deab9266-d7c5-4886-a2b5-6691569a4134-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:05.922450 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922324 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deab9266-d7c5-4886-a2b5-6691569a4134-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:05.922450 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922357 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:05.922450 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922380 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/deab9266-d7c5-4886-a2b5-6691569a4134-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:05.922450 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922405 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deab9266-d7c5-4886-a2b5-6691569a4134-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:05.922660 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922457 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6wq7\" (UniqueName: \"kubernetes.io/projected/7b37e713-d8f4-4a88-9c98-414a2c310553-kube-api-access-j6wq7\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:05.922660 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922482 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqm9k\" (UniqueName: \"kubernetes.io/projected/0950289d-133e-441f-9ea8-ccc434f1c101-kube-api-access-qqm9k\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:05.922660 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922506 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b37e713-d8f4-4a88-9c98-414a2c310553-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:05.922660 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922541 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0950289d-133e-441f-9ea8-ccc434f1c101-metrics-client-ca\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:05.922660 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922636 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/deab9266-d7c5-4886-a2b5-6691569a4134-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:05.922890 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922679 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xllz\" (UniqueName: \"kubernetes.io/projected/deab9266-d7c5-4886-a2b5-6691569a4134-kube-api-access-8xllz\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:05.922890 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922714 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-accelerators-collector-config\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:05.922890 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922767 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b37e713-d8f4-4a88-9c98-414a2c310553-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:05.922890 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922802 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b37e713-d8f4-4a88-9c98-414a2c310553-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:05.922890 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:05.922826 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0950289d-133e-441f-9ea8-ccc434f1c101-root\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.023540 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023509 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6wq7\" (UniqueName: \"kubernetes.io/projected/7b37e713-d8f4-4a88-9c98-414a2c310553-kube-api-access-j6wq7\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:06.023540 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023539 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqm9k\" (UniqueName: \"kubernetes.io/projected/0950289d-133e-441f-9ea8-ccc434f1c101-kube-api-access-qqm9k\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.023773 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023559 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b37e713-d8f4-4a88-9c98-414a2c310553-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:06.023773 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023605 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0950289d-133e-441f-9ea8-ccc434f1c101-metrics-client-ca\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.023773 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023646 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/deab9266-d7c5-4886-a2b5-6691569a4134-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.023773 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023678 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xllz\" (UniqueName: \"kubernetes.io/projected/deab9266-d7c5-4886-a2b5-6691569a4134-kube-api-access-8xllz\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.023773 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023729 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-accelerators-collector-config\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024033 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023760 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b37e713-d8f4-4a88-9c98-414a2c310553-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:06.024033 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023813 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b37e713-d8f4-4a88-9c98-414a2c310553-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:06.024033 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023881 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0950289d-133e-441f-9ea8-ccc434f1c101-root\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024033 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023941 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-wtmp\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024033 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.023986 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-tls\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024033 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024013 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0950289d-133e-441f-9ea8-ccc434f1c101-sys\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024052 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-textfile\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024086 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/deab9266-d7c5-4886-a2b5-6691569a4134-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.024316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024115 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deab9266-d7c5-4886-a2b5-6691569a4134-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.024316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024125 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-wtmp\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024148 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024175 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/deab9266-d7c5-4886-a2b5-6691569a4134-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.024316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024180 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0950289d-133e-441f-9ea8-ccc434f1c101-root\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024198 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deab9266-d7c5-4886-a2b5-6691569a4134-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.024316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024304 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0950289d-133e-441f-9ea8-ccc434f1c101-metrics-client-ca\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024826 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024358 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0950289d-133e-441f-9ea8-ccc434f1c101-sys\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024826 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024448 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/deab9266-d7c5-4886-a2b5-6691569a4134-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.024826 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024694 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/deab9266-d7c5-4886-a2b5-6691569a4134-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.024826 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024715 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-textfile\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.024826 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024764 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b37e713-d8f4-4a88-9c98-414a2c310553-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:06.025060 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.024880 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deab9266-d7c5-4886-a2b5-6691569a4134-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.025384 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.025341 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-accelerators-collector-config\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.026865 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.026837 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b37e713-d8f4-4a88-9c98-414a2c310553-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:06.027108 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.027077 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b37e713-d8f4-4a88-9c98-414a2c310553-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:06.027728 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.027705 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/deab9266-d7c5-4886-a2b5-6691569a4134-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.027824 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.027705 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-tls\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.027824 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.027702 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deab9266-d7c5-4886-a2b5-6691569a4134-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.027824 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.027715 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0950289d-133e-441f-9ea8-ccc434f1c101-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.031146 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.031114 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqm9k\" (UniqueName: \"kubernetes.io/projected/0950289d-133e-441f-9ea8-ccc434f1c101-kube-api-access-qqm9k\") pod \"node-exporter-4ckfg\" (UID: \"0950289d-133e-441f-9ea8-ccc434f1c101\") " pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.031290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.031269 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6wq7\" (UniqueName: \"kubernetes.io/projected/7b37e713-d8f4-4a88-9c98-414a2c310553-kube-api-access-j6wq7\") pod \"openshift-state-metrics-67dcf8c979-drs75\" (UID: \"7b37e713-d8f4-4a88-9c98-414a2c310553\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:06.031890 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.031874 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xllz\" (UniqueName: \"kubernetes.io/projected/deab9266-d7c5-4886-a2b5-6691569a4134-kube-api-access-8xllz\") pod \"kube-state-metrics-59fbdd77d-vqz77\" (UID: \"deab9266-d7c5-4886-a2b5-6691569a4134\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.084829 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.084806 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" Sep 17 01:42:06.096203 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.096172 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4ckfg" Sep 17 01:42:06.103213 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.103177 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" Sep 17 01:42:06.107525 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:42:06.107506 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0950289d_133e_441f_9ea8_ccc434f1c101.slice/crio-619224b9854413b40a976bcbd6f64700c7e1cdef65126818e5668624cf49bfed WatchSource:0}: Error finding container 619224b9854413b40a976bcbd6f64700c7e1cdef65126818e5668624cf49bfed: Status 404 returned error can't find the container with id 619224b9854413b40a976bcbd6f64700c7e1cdef65126818e5668624cf49bfed Sep 17 01:42:06.362013 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.361992 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75"] Sep 17 01:42:06.364111 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:42:06.364085 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b37e713_d8f4_4a88_9c98_414a2c310553.slice/crio-8d3c887f604bd453380367098fa4f9666b3cb368a690c0cfde97740b4c460e4e WatchSource:0}: Error finding container 8d3c887f604bd453380367098fa4f9666b3cb368a690c0cfde97740b4c460e4e: Status 404 returned error can't find the container with id 8d3c887f604bd453380367098fa4f9666b3cb368a690c0cfde97740b4c460e4e Sep 17 01:42:06.390393 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.390367 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77"] Sep 17 01:42:06.401537 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:42:06.401509 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeab9266_d7c5_4886_a2b5_6691569a4134.slice/crio-f7422d2554c069006524f4f2f3d4d6e8a0abb16c9367050e47150b13b630680a WatchSource:0}: Error finding container f7422d2554c069006524f4f2f3d4d6e8a0abb16c9367050e47150b13b630680a: Status 404 returned error can't find the container with id f7422d2554c069006524f4f2f3d4d6e8a0abb16c9367050e47150b13b630680a Sep 17 01:42:06.709821 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.709727 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" event={"ID":"deab9266-d7c5-4886-a2b5-6691569a4134","Type":"ContainerStarted","Data":"f7422d2554c069006524f4f2f3d4d6e8a0abb16c9367050e47150b13b630680a"} Sep 17 01:42:06.712477 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.712367 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" event={"ID":"7b37e713-d8f4-4a88-9c98-414a2c310553","Type":"ContainerStarted","Data":"c06154f5e00800e02686de04dce639e9143369b48c7d42c0802f297d96b2e017"} Sep 17 01:42:06.712477 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.712424 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" event={"ID":"7b37e713-d8f4-4a88-9c98-414a2c310553","Type":"ContainerStarted","Data":"fcd5a9a8bfc8b55a3997dec5d18fb5a5d73b7891f5a5c9b1633acc4f2a0b6682"} Sep 17 01:42:06.712477 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.712454 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" event={"ID":"7b37e713-d8f4-4a88-9c98-414a2c310553","Type":"ContainerStarted","Data":"8d3c887f604bd453380367098fa4f9666b3cb368a690c0cfde97740b4c460e4e"} Sep 17 01:42:06.715286 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.715234 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4ckfg" event={"ID":"0950289d-133e-441f-9ea8-ccc434f1c101","Type":"ContainerStarted","Data":"619224b9854413b40a976bcbd6f64700c7e1cdef65126818e5668624cf49bfed"} Sep 17 01:42:06.720737 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.720714 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pb4gq" event={"ID":"ac42c232-24c7-4609-a169-07fe9bb4c2e6","Type":"ContainerStarted","Data":"d0794da35e386000d8b7be651d841afdd64fb0d6f9a922e22e410b56f90a6624"} Sep 17 01:42:06.720933 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.720917 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pb4gq" event={"ID":"ac42c232-24c7-4609-a169-07fe9bb4c2e6","Type":"ContainerStarted","Data":"5a4185452236bf93903c6b1726d71ec699474f268137cd98e4aa2850dd234dae"} Sep 17 01:42:06.880298 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.880250 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pb4gq" podStartSLOduration=252.044268925 podStartE2EDuration="4m13.880231812s" totalImagesPullingTime="1.835962887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:06.738491588 +0000 UTC m=+290.495258850" watchObservedRunningTime="2026-09-17 01:42:06.880231812 +0000 UTC m=+290.636999055" Sep 17 01:42:06.880432 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.880393 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:42:06.884014 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.883998 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:06.886381 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.886348 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 01:42:06.886873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.886563 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 01:42:06.886873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.886598 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 01:42:06.886873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.886626 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 01:42:06.886873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.886633 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 01:42:06.886873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.886563 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 01:42:06.886873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.886773 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 01:42:06.886873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.886778 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 01:42:06.886873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.886785 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gbbn4\"" Sep 17 01:42:06.886873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.886802 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 01:42:06.899181 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:06.899159 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:42:07.032562 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032487 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.032562 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032531 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bz8v\" (UniqueName: \"kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-kube-api-access-8bz8v\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.032791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032563 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.032791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032620 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.032791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032660 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.032791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032703 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-web-config\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.032791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032728 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.032791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032743 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.032791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032771 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.033056 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032797 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.033056 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032835 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.033056 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032857 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-config-out\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.033056 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.032930 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-config-volume\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.133559 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133529 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-web-config\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.133755 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133565 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.133755 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133606 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.133755 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133633 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.133755 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133660 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.133755 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133690 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.133755 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133711 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-config-out\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.134071 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133761 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-config-volume\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.134071 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133801 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.134071 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133824 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bz8v\" (UniqueName: \"kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-kube-api-access-8bz8v\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.134071 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133843 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.134071 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133861 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.134071 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.133906 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.134824 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.134485 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.135004 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.134982 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.135637 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.135607 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.137350 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.137135 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.137350 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.137288 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.137350 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.137312 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-config-out\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.137554 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.137390 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-web-config\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.138352 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.138314 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.138787 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.138662 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.138914 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.138893 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.138978 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.138928 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.139757 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.139738 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-config-volume\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.142562 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.142542 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bz8v\" (UniqueName: \"kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-kube-api-access-8bz8v\") pod \"alertmanager-main-0\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.194153 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.194127 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:42:07.479411 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.479364 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:42:07.483224 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:42:07.483180 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec1754e_c729_4793_b678_f8ab1274cafa.slice/crio-ae0aff23ae39acbd9fc41ece70436851d1abf4bf5420173258860793f79010db WatchSource:0}: Error finding container ae0aff23ae39acbd9fc41ece70436851d1abf4bf5420173258860793f79010db: Status 404 returned error can't find the container with id ae0aff23ae39acbd9fc41ece70436851d1abf4bf5420173258860793f79010db Sep 17 01:42:07.726744 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.726701 2586 generic.go:356] "Generic (PLEG): container finished" podID="0950289d-133e-441f-9ea8-ccc434f1c101" containerID="ed9193c79414057e95b9d9909e31a5c29295b8383b44e7eabd17d7d1a481b5a1" exitCode=0 Sep 17 01:42:07.727205 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.726787 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4ckfg" event={"ID":"0950289d-133e-441f-9ea8-ccc434f1c101","Type":"ContainerDied","Data":"ed9193c79414057e95b9d9909e31a5c29295b8383b44e7eabd17d7d1a481b5a1"} Sep 17 01:42:07.728649 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.728621 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerStarted","Data":"ae0aff23ae39acbd9fc41ece70436851d1abf4bf5420173258860793f79010db"} Sep 17 01:42:07.728894 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:07.728873 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pb4gq" Sep 17 01:42:08.735988 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:08.735897 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" event={"ID":"deab9266-d7c5-4886-a2b5-6691569a4134","Type":"ContainerStarted","Data":"cb4271b5c2291bab99f629d4c20e9b32851f1118e7831b25949e28c9e04c3efd"} Sep 17 01:42:08.736413 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:08.736000 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" event={"ID":"deab9266-d7c5-4886-a2b5-6691569a4134","Type":"ContainerStarted","Data":"3c5a7550b5bc02fae06043523992c42104860b5ecffce8e46937934f97fb5df1"} Sep 17 01:42:08.736413 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:08.736036 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" event={"ID":"deab9266-d7c5-4886-a2b5-6691569a4134","Type":"ContainerStarted","Data":"e37afbe85e29947cfd979743dac66b26a20b3805ab3aa601484706714e54c21f"} Sep 17 01:42:08.738680 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:08.738648 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" event={"ID":"7b37e713-d8f4-4a88-9c98-414a2c310553","Type":"ContainerStarted","Data":"69427e5bbd8cb21f94258a2283aeed619217e007d6c3d3f1608eea423eadf5ee"} Sep 17 01:42:08.741184 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:08.741155 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4ckfg" event={"ID":"0950289d-133e-441f-9ea8-ccc434f1c101","Type":"ContainerStarted","Data":"01d08d897478b414c0828c3e4cda2786b8c8c26bfbf1ba85ebaa9abf4244afe7"} Sep 17 01:42:08.741308 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:08.741208 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4ckfg" event={"ID":"0950289d-133e-441f-9ea8-ccc434f1c101","Type":"ContainerStarted","Data":"dcc39012bfb9f3de59d86afd4532df57f45c4e6e735802bf1d58e165151ccd3d"} Sep 17 01:42:08.758985 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:08.758938 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-vqz77" podStartSLOduration=2.044735551 podStartE2EDuration="3.758921328s" totalImagesPullingTime="1.714185777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:08.757538113 +0000 UTC m=+292.514305372" watchObservedRunningTime="2026-09-17 01:42:08.758921328 +0000 UTC m=+292.515688572" Sep 17 01:42:08.782151 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:08.782084 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4ckfg" podStartSLOduration=2.668035273 podStartE2EDuration="3.782067763s" totalImagesPullingTime="1.030947387s" totalInitContainerRuntime="83.085103ms" isStatefulPod=true podCreationTimestamp="2026-09-17 01:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:08.781467068 +0000 UTC m=+292.538234333" watchObservedRunningTime="2026-09-17 01:42:08.782067763 +0000 UTC m=+292.538835007" Sep 17 01:42:08.798171 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:08.798135 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-drs75" podStartSLOduration=2.179767333 podStartE2EDuration="3.798122508s" totalImagesPullingTime="1.618355175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:08.797466545 +0000 UTC m=+292.554233808" watchObservedRunningTime="2026-09-17 01:42:08.798122508 +0000 UTC m=+292.554889751" Sep 17 01:42:09.746308 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:09.746272 2586 generic.go:356] "Generic (PLEG): container finished" podID="aec1754e-c729-4793-b678-f8ab1274cafa" containerID="4cff21b6063ac76c5a1f14eadffd09a865b531d354c485fc3e6ac01bb2bc076b" exitCode=0 Sep 17 01:42:09.746772 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:09.746399 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerDied","Data":"4cff21b6063ac76c5a1f14eadffd09a865b531d354c485fc3e6ac01bb2bc076b"} Sep 17 01:42:10.158980 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.158943 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bc54fcff7-8c987"] Sep 17 01:42:10.163414 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.163392 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.165824 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.165798 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 17 01:42:10.165927 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.165845 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dkx6x\"" Sep 17 01:42:10.166638 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.166620 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 17 01:42:10.166771 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.166652 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 17 01:42:10.166771 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.166701 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bli2mhoa5lqn3\"" Sep 17 01:42:10.166771 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.166712 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 01:42:10.173808 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.173783 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bc54fcff7-8c987"] Sep 17 01:42:10.264007 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.263974 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8e594e9e-2526-41cd-9a11-c44e0dca792c-secret-metrics-server-tls\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.264131 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.264033 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8e594e9e-2526-41cd-9a11-c44e0dca792c-secret-metrics-server-client-certs\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.264131 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.264088 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e594e9e-2526-41cd-9a11-c44e0dca792c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.264131 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.264124 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8e594e9e-2526-41cd-9a11-c44e0dca792c-metrics-server-audit-profiles\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.264246 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.264155 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8e594e9e-2526-41cd-9a11-c44e0dca792c-audit-log\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.264246 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.264173 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxnj9\" (UniqueName: \"kubernetes.io/projected/8e594e9e-2526-41cd-9a11-c44e0dca792c-kube-api-access-zxnj9\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.264246 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.264231 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8e594e9e-2526-41cd-9a11-c44e0dca792c-client-ca-bundle\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.365249 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.365218 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8e594e9e-2526-41cd-9a11-c44e0dca792c-client-ca-bundle\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.365442 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.365267 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8e594e9e-2526-41cd-9a11-c44e0dca792c-secret-metrics-server-tls\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.365442 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.365316 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8e594e9e-2526-41cd-9a11-c44e0dca792c-secret-metrics-server-client-certs\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.365442 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.365347 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e594e9e-2526-41cd-9a11-c44e0dca792c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.365442 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.365388 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8e594e9e-2526-41cd-9a11-c44e0dca792c-metrics-server-audit-profiles\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.365442 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.365420 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8e594e9e-2526-41cd-9a11-c44e0dca792c-audit-log\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.365715 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.365445 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxnj9\" (UniqueName: \"kubernetes.io/projected/8e594e9e-2526-41cd-9a11-c44e0dca792c-kube-api-access-zxnj9\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.366205 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.366166 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8e594e9e-2526-41cd-9a11-c44e0dca792c-audit-log\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.366908 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.366881 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e594e9e-2526-41cd-9a11-c44e0dca792c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.367369 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.367342 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8e594e9e-2526-41cd-9a11-c44e0dca792c-metrics-server-audit-profiles\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.368603 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.368556 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8e594e9e-2526-41cd-9a11-c44e0dca792c-secret-metrics-server-client-certs\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.368686 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.368649 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8e594e9e-2526-41cd-9a11-c44e0dca792c-client-ca-bundle\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.368746 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.368700 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8e594e9e-2526-41cd-9a11-c44e0dca792c-secret-metrics-server-tls\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.373123 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.373085 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxnj9\" (UniqueName: \"kubernetes.io/projected/8e594e9e-2526-41cd-9a11-c44e0dca792c-kube-api-access-zxnj9\") pod \"metrics-server-bc54fcff7-8c987\" (UID: \"8e594e9e-2526-41cd-9a11-c44e0dca792c\") " pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.478823 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.478721 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:10.767452 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:10.767422 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bc54fcff7-8c987"] Sep 17 01:42:10.769571 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:42:10.769545 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e594e9e_2526_41cd_9a11_c44e0dca792c.slice/crio-72b9dcb66534f4ad991475a9bfc880a757dff5fd84d1ea2686f5f2beba62f92f WatchSource:0}: Error finding container 72b9dcb66534f4ad991475a9bfc880a757dff5fd84d1ea2686f5f2beba62f92f: Status 404 returned error can't find the container with id 72b9dcb66534f4ad991475a9bfc880a757dff5fd84d1ea2686f5f2beba62f92f Sep 17 01:42:11.759655 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:11.759615 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerStarted","Data":"6a94f348af71a513ba84aa7b7d3ca87381804145139085d6efcf436ad7c58a14"} Sep 17 01:42:11.759830 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:11.759706 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerStarted","Data":"22518704b182404f441b24e5af3967da010c497a11638b946869b80a698f5535"} Sep 17 01:42:11.759830 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:11.759753 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerStarted","Data":"cdf0eb04f74bc2d6b27f7619f7dabc8969d038c78bfd3aa71ae47a0c592ca9bf"} Sep 17 01:42:11.759830 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:11.759787 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerStarted","Data":"04f451531bb66aa90c76758a6ac4ab2b507dca3fe9cff694a2b6f1319bf140cb"} Sep 17 01:42:11.759830 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:11.759823 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerStarted","Data":"d7e68dda092a4d5d7e5433e8351a1097a925b7f1641ead66f45557e65ba646dd"} Sep 17 01:42:11.761030 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:11.761008 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" event={"ID":"8e594e9e-2526-41cd-9a11-c44e0dca792c","Type":"ContainerStarted","Data":"72b9dcb66534f4ad991475a9bfc880a757dff5fd84d1ea2686f5f2beba62f92f"} Sep 17 01:42:12.768599 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:12.768546 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerStarted","Data":"fca2034707ba2a81a801992c6303c99e005bcc118f71feb21d965081c3014875"} Sep 17 01:42:12.770039 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:12.770015 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" event={"ID":"8e594e9e-2526-41cd-9a11-c44e0dca792c","Type":"ContainerStarted","Data":"5f570a4f654aa0dea9d966bdeabe28ac1d834dfa231387a4b5d03fd5b4dc75a1"} Sep 17 01:42:12.794825 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:12.794783 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.975723555 podStartE2EDuration="6.794766219s" totalImagesPullingTime="3.799577726s" totalInitContainerRuntime="19.464938ms" isStatefulPod=false podCreationTimestamp="2026-09-17 01:42:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:12.794690737 +0000 UTC m=+296.551458000" watchObservedRunningTime="2026-09-17 01:42:12.794766219 +0000 UTC m=+296.551533463" Sep 17 01:42:12.811516 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:12.811436 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" podStartSLOduration=1.336800049 podStartE2EDuration="2.811422953s" totalImagesPullingTime="1.474622904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:12.810009306 +0000 UTC m=+296.566776567" watchObservedRunningTime="2026-09-17 01:42:12.811422953 +0000 UTC m=+296.568190232" Sep 17 01:42:13.762632 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:42:13.762563 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-bjxzr" podUID="26801bb3-4b53-4677-a222-d71afd225350" Sep 17 01:42:16.689907 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:16.689881 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:42:16.690307 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:16.690004 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:42:16.701287 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:16.701271 2586 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 01:42:16.763720 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:16.763571 2586 scope.go:117] "RemoveContainer" containerID="2599d224b492a4e85715acf4de62e39ba20b973e2042338661e93022522326d4" Sep 17 01:42:17.743671 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:17.743631 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pb4gq" Sep 17 01:42:17.790472 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:17.790446 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:42:17.790694 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:17.790534 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" event={"ID":"d084eabd-5582-43e0-84bf-d5a860eb8092","Type":"ContainerStarted","Data":"107b47ea4e81019d0b619eaebca0141b0aa6d9a2ceabde80e1b6e9fb20371ed9"} Sep 17 01:42:17.790904 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:17.790882 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:42:17.795842 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:17.795821 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" Sep 17 01:42:17.808210 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:17.808141 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-xz5g9" podStartSLOduration=52.733553632 podStartE2EDuration="55.808125724s" totalImagesPullingTime="3.074572092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:17.807241572 +0000 UTC m=+301.564008808" watchObservedRunningTime="2026-09-17 01:42:17.808125724 +0000 UTC m=+301.564892995" Sep 17 01:42:23.339097 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:23.339061 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c6587d777-q8lc4"] Sep 17 01:42:23.343700 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:23.343666 2586 patch_prober.go:28] interesting pod/image-registry-6c6587d777-q8lc4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 17 01:42:23.343829 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:23.343715 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" podUID="d8ed7b30-477c-480e-888f-94a13fb9892d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 01:42:28.761294 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:28.761255 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:42:30.479691 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:30.479655 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:30.480107 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:30.479718 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:33.343979 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:33.343951 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:42:37.868714 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:37.868676 2586 generic.go:356] "Generic (PLEG): container finished" podID="3ca692a6-2629-40fb-a8fc-26571248db38" containerID="53613ba4f8249b0913221f5e1431e9d245faa1217604a1e40b693ec15e608378" exitCode=0 Sep 17 01:42:37.869082 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:37.868754 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" event={"ID":"3ca692a6-2629-40fb-a8fc-26571248db38","Type":"ContainerDied","Data":"53613ba4f8249b0913221f5e1431e9d245faa1217604a1e40b693ec15e608378"} Sep 17 01:42:37.869122 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:37.869087 2586 scope.go:117] "RemoveContainer" containerID="53613ba4f8249b0913221f5e1431e9d245faa1217604a1e40b693ec15e608378" Sep 17 01:42:38.874803 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:38.874766 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w9lgf" event={"ID":"3ca692a6-2629-40fb-a8fc-26571248db38","Type":"ContainerStarted","Data":"8cbf2e20cba0b1b906a2ffe83448eb9c092376dcd7a152e798c2c9332b4143d1"} Sep 17 01:42:42.892124 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:42.892089 2586 generic.go:356] "Generic (PLEG): container finished" podID="2a42508a-cdab-4526-8b1a-de7088e24b00" containerID="657e88db405c5e932f13fc5a5a13b5b194374191c688ef78949386add4d24cd8" exitCode=0 Sep 17 01:42:42.892485 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:42.892166 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dcgdm" event={"ID":"2a42508a-cdab-4526-8b1a-de7088e24b00","Type":"ContainerDied","Data":"657e88db405c5e932f13fc5a5a13b5b194374191c688ef78949386add4d24cd8"} Sep 17 01:42:42.892526 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:42.892502 2586 scope.go:117] "RemoveContainer" containerID="657e88db405c5e932f13fc5a5a13b5b194374191c688ef78949386add4d24cd8" Sep 17 01:42:43.897885 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:43.897855 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dcgdm" event={"ID":"2a42508a-cdab-4526-8b1a-de7088e24b00","Type":"ContainerStarted","Data":"ae56267dbfe88b91a4351358685ff7d1d509acd1ee1dbfa9b01d018577d1c30f"} Sep 17 01:42:48.358694 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.358646 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" podUID="d8ed7b30-477c-480e-888f-94a13fb9892d" containerName="registry" containerID="cri-o://d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6" gracePeriod=30 Sep 17 01:42:48.610188 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.610109 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:42:48.689817 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.689787 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") pod \"d8ed7b30-477c-480e-888f-94a13fb9892d\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " Sep 17 01:42:48.690001 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.689827 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8ed7b30-477c-480e-888f-94a13fb9892d-ca-trust-extracted\") pod \"d8ed7b30-477c-480e-888f-94a13fb9892d\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " Sep 17 01:42:48.690001 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.689902 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwlsr\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-kube-api-access-rwlsr\") pod \"d8ed7b30-477c-480e-888f-94a13fb9892d\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " Sep 17 01:42:48.690001 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.689946 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-trusted-ca\") pod \"d8ed7b30-477c-480e-888f-94a13fb9892d\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " Sep 17 01:42:48.690001 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.689968 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-bound-sa-token\") pod \"d8ed7b30-477c-480e-888f-94a13fb9892d\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " Sep 17 01:42:48.690001 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.689997 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-certificates\") pod \"d8ed7b30-477c-480e-888f-94a13fb9892d\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " Sep 17 01:42:48.690240 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.690035 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-image-registry-private-configuration\") pod \"d8ed7b30-477c-480e-888f-94a13fb9892d\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " Sep 17 01:42:48.690240 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.690063 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-installation-pull-secrets\") pod \"d8ed7b30-477c-480e-888f-94a13fb9892d\" (UID: \"d8ed7b30-477c-480e-888f-94a13fb9892d\") " Sep 17 01:42:48.690492 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.690456 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d8ed7b30-477c-480e-888f-94a13fb9892d" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:42:48.690679 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.690647 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d8ed7b30-477c-480e-888f-94a13fb9892d" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:42:48.692805 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.692752 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d8ed7b30-477c-480e-888f-94a13fb9892d" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:42:48.692905 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.692873 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d8ed7b30-477c-480e-888f-94a13fb9892d" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:42:48.692959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.692931 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d8ed7b30-477c-480e-888f-94a13fb9892d" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:42:48.693010 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.692969 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-kube-api-access-rwlsr" (OuterVolumeSpecName: "kube-api-access-rwlsr") pod "d8ed7b30-477c-480e-888f-94a13fb9892d" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d"). InnerVolumeSpecName "kube-api-access-rwlsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:42:48.693010 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.692967 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d8ed7b30-477c-480e-888f-94a13fb9892d" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:42:48.698457 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.698435 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ed7b30-477c-480e-888f-94a13fb9892d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d8ed7b30-477c-480e-888f-94a13fb9892d" (UID: "d8ed7b30-477c-480e-888f-94a13fb9892d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:42:48.791755 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.791721 2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-trusted-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:42:48.791755 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.791748 2586 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-bound-sa-token\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:42:48.791755 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.791757 2586 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-certificates\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:42:48.791975 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.791768 2586 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-image-registry-private-configuration\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:42:48.791975 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.791780 2586 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8ed7b30-477c-480e-888f-94a13fb9892d-installation-pull-secrets\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:42:48.791975 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.791788 2586 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-registry-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:42:48.791975 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.791797 2586 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8ed7b30-477c-480e-888f-94a13fb9892d-ca-trust-extracted\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:42:48.791975 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.791805 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwlsr\" (UniqueName: \"kubernetes.io/projected/d8ed7b30-477c-480e-888f-94a13fb9892d-kube-api-access-rwlsr\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:42:48.917759 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.917669 2586 generic.go:356] "Generic (PLEG): container finished" podID="d8ed7b30-477c-480e-888f-94a13fb9892d" containerID="d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6" exitCode=0 Sep 17 01:42:48.917759 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.917725 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" Sep 17 01:42:48.917924 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.917764 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" event={"ID":"d8ed7b30-477c-480e-888f-94a13fb9892d","Type":"ContainerDied","Data":"d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6"} Sep 17 01:42:48.917924 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.917815 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c6587d777-q8lc4" event={"ID":"d8ed7b30-477c-480e-888f-94a13fb9892d","Type":"ContainerDied","Data":"f3c6205337183368ad90fe6029b32264f7fda38597b4fcd14eef7acb7b3ea497"} Sep 17 01:42:48.917924 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.917843 2586 scope.go:117] "RemoveContainer" containerID="d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6" Sep 17 01:42:48.926527 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.926503 2586 scope.go:117] "RemoveContainer" containerID="d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6" Sep 17 01:42:48.926836 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:42:48.926809 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6\": container with ID starting with d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6 not found: ID does not exist" containerID="d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6" Sep 17 01:42:48.926887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.926845 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6"} err="failed to get container status \"d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6\": rpc error: code = NotFound desc = could not find container \"d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6\": container with ID starting with d005ad2b55a621dd9c9d3c5588bf357838a160e77ba5c9cb0f3e730eb62cdec6 not found: ID does not exist" Sep 17 01:42:48.933007 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.932985 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c6587d777-q8lc4"] Sep 17 01:42:48.936558 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:48.936536 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c6587d777-q8lc4"] Sep 17 01:42:50.484952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:50.484924 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:50.488823 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:50.488799 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bc54fcff7-8c987" Sep 17 01:42:50.766359 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:42:50.766268 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8ed7b30-477c-480e-888f-94a13fb9892d" path="/var/lib/kubelet/pods/d8ed7b30-477c-480e-888f-94a13fb9892d/volumes" Sep 17 01:43:25.935975 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:25.935935 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:43:25.936476 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:25.936245 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="alertmanager" containerID="cri-o://d7e68dda092a4d5d7e5433e8351a1097a925b7f1641ead66f45557e65ba646dd" gracePeriod=120 Sep 17 01:43:25.936476 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:25.936310 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy-metric" containerID="cri-o://6a94f348af71a513ba84aa7b7d3ca87381804145139085d6efcf436ad7c58a14" gracePeriod=120 Sep 17 01:43:25.936476 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:25.936353 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy-web" containerID="cri-o://cdf0eb04f74bc2d6b27f7619f7dabc8969d038c78bfd3aa71ae47a0c592ca9bf" gracePeriod=120 Sep 17 01:43:25.936476 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:25.936369 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy" containerID="cri-o://22518704b182404f441b24e5af3967da010c497a11638b946869b80a698f5535" gracePeriod=120 Sep 17 01:43:25.936476 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:25.936396 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="prom-label-proxy" containerID="cri-o://fca2034707ba2a81a801992c6303c99e005bcc118f71feb21d965081c3014875" gracePeriod=120 Sep 17 01:43:25.936745 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:25.936546 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="config-reloader" containerID="cri-o://04f451531bb66aa90c76758a6ac4ab2b507dca3fe9cff694a2b6f1319bf140cb" gracePeriod=120 Sep 17 01:43:26.062551 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:26.062520 2586 generic.go:356] "Generic (PLEG): container finished" podID="aec1754e-c729-4793-b678-f8ab1274cafa" containerID="fca2034707ba2a81a801992c6303c99e005bcc118f71feb21d965081c3014875" exitCode=0 Sep 17 01:43:26.062551 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:26.062547 2586 generic.go:356] "Generic (PLEG): container finished" podID="aec1754e-c729-4793-b678-f8ab1274cafa" containerID="22518704b182404f441b24e5af3967da010c497a11638b946869b80a698f5535" exitCode=0 Sep 17 01:43:26.062551 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:26.062554 2586 generic.go:356] "Generic (PLEG): container finished" podID="aec1754e-c729-4793-b678-f8ab1274cafa" containerID="04f451531bb66aa90c76758a6ac4ab2b507dca3fe9cff694a2b6f1319bf140cb" exitCode=0 Sep 17 01:43:26.062778 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:26.062560 2586 generic.go:356] "Generic (PLEG): container finished" podID="aec1754e-c729-4793-b678-f8ab1274cafa" containerID="d7e68dda092a4d5d7e5433e8351a1097a925b7f1641ead66f45557e65ba646dd" exitCode=0 Sep 17 01:43:26.062778 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:26.062611 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerDied","Data":"fca2034707ba2a81a801992c6303c99e005bcc118f71feb21d965081c3014875"} Sep 17 01:43:26.062778 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:26.062682 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerDied","Data":"22518704b182404f441b24e5af3967da010c497a11638b946869b80a698f5535"} Sep 17 01:43:26.062778 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:26.062708 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerDied","Data":"04f451531bb66aa90c76758a6ac4ab2b507dca3fe9cff694a2b6f1319bf140cb"} Sep 17 01:43:26.062778 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:26.062733 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerDied","Data":"d7e68dda092a4d5d7e5433e8351a1097a925b7f1641ead66f45557e65ba646dd"} Sep 17 01:43:27.074727 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.074686 2586 generic.go:356] "Generic (PLEG): container finished" podID="aec1754e-c729-4793-b678-f8ab1274cafa" containerID="6a94f348af71a513ba84aa7b7d3ca87381804145139085d6efcf436ad7c58a14" exitCode=0 Sep 17 01:43:27.074727 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.074722 2586 generic.go:356] "Generic (PLEG): container finished" podID="aec1754e-c729-4793-b678-f8ab1274cafa" containerID="cdf0eb04f74bc2d6b27f7619f7dabc8969d038c78bfd3aa71ae47a0c592ca9bf" exitCode=0 Sep 17 01:43:27.075138 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.074758 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerDied","Data":"6a94f348af71a513ba84aa7b7d3ca87381804145139085d6efcf436ad7c58a14"} Sep 17 01:43:27.075138 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.074857 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerDied","Data":"cdf0eb04f74bc2d6b27f7619f7dabc8969d038c78bfd3aa71ae47a0c592ca9bf"} Sep 17 01:43:27.209568 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.209541 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:27.317918 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.317819 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-web-config\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.317918 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.317858 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.317918 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.317878 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-config-volume\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.317918 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.317920 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-config-out\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.318251 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.317948 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-main-db\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.318251 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.317986 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bz8v\" (UniqueName: \"kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-kube-api-access-8bz8v\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.318251 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.318025 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-cluster-tls-config\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.318251 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.318056 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-trusted-ca-bundle\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.318251 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.318116 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-main-tls\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.318251 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.318138 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-metrics-client-ca\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.318251 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.318168 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.318251 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.318193 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-web\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.318251 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.318222 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-tls-assets\") pod \"aec1754e-c729-4793-b678-f8ab1274cafa\" (UID: \"aec1754e-c729-4793-b678-f8ab1274cafa\") " Sep 17 01:43:27.318891 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.318336 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:43:27.318891 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.318505 2586 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-main-db\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.318891 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.318763 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:43:27.319376 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.319345 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:43:27.320985 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.320956 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-config-out" (OuterVolumeSpecName: "config-out") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:43:27.321354 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.321309 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-kube-api-access-8bz8v" (OuterVolumeSpecName: "kube-api-access-8bz8v") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "kube-api-access-8bz8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:43:27.321459 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.321378 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-config-volume" (OuterVolumeSpecName: "config-volume") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:27.321526 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.321488 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:27.321701 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.321670 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:43:27.321959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.321932 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:27.322908 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.322883 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:27.323427 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.323408 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:27.326157 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.326124 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:27.333136 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.333110 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-web-config" (OuterVolumeSpecName: "web-config") pod "aec1754e-c729-4793-b678-f8ab1274cafa" (UID: "aec1754e-c729-4793-b678-f8ab1274cafa"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:27.419745 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419696 2586 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aec1754e-c729-4793-b678-f8ab1274cafa-config-out\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419745 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419739 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bz8v\" (UniqueName: \"kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-kube-api-access-8bz8v\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419745 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419751 2586 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-cluster-tls-config\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419745 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419761 2586 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419771 2586 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-main-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419780 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aec1754e-c729-4793-b678-f8ab1274cafa-metrics-client-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419789 2586 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419799 2586 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419807 2586 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aec1754e-c729-4793-b678-f8ab1274cafa-tls-assets\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419814 2586 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-web-config\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419822 2586 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:27.419996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:27.419830 2586 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aec1754e-c729-4793-b678-f8ab1274cafa-config-volume\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:43:28.082294 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.082249 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aec1754e-c729-4793-b678-f8ab1274cafa","Type":"ContainerDied","Data":"ae0aff23ae39acbd9fc41ece70436851d1abf4bf5420173258860793f79010db"} Sep 17 01:43:28.082294 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.082276 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.082736 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.082332 2586 scope.go:117] "RemoveContainer" containerID="fca2034707ba2a81a801992c6303c99e005bcc118f71feb21d965081c3014875" Sep 17 01:43:28.091031 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.091012 2586 scope.go:117] "RemoveContainer" containerID="6a94f348af71a513ba84aa7b7d3ca87381804145139085d6efcf436ad7c58a14" Sep 17 01:43:28.099057 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.099037 2586 scope.go:117] "RemoveContainer" containerID="22518704b182404f441b24e5af3967da010c497a11638b946869b80a698f5535" Sep 17 01:43:28.105379 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.105355 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:43:28.107256 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.107232 2586 scope.go:117] "RemoveContainer" containerID="cdf0eb04f74bc2d6b27f7619f7dabc8969d038c78bfd3aa71ae47a0c592ca9bf" Sep 17 01:43:28.109438 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.109418 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:43:28.114723 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.114705 2586 scope.go:117] "RemoveContainer" containerID="04f451531bb66aa90c76758a6ac4ab2b507dca3fe9cff694a2b6f1319bf140cb" Sep 17 01:43:28.121988 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.121966 2586 scope.go:117] "RemoveContainer" containerID="d7e68dda092a4d5d7e5433e8351a1097a925b7f1641ead66f45557e65ba646dd" Sep 17 01:43:28.129922 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.129902 2586 scope.go:117] "RemoveContainer" containerID="4cff21b6063ac76c5a1f14eadffd09a865b531d354c485fc3e6ac01bb2bc076b" Sep 17 01:43:28.135676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135651 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:43:28.135932 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135921 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy" Sep 17 01:43:28.135970 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135936 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy" Sep 17 01:43:28.135970 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135946 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="config-reloader" Sep 17 01:43:28.135970 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135951 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="config-reloader" Sep 17 01:43:28.135970 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135960 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="init-config-reloader" Sep 17 01:43:28.135970 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135966 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="init-config-reloader" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135973 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy-metric" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135979 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy-metric" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135990 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="alertmanager" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.135995 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="alertmanager" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136000 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy-web" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136004 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy-web" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136011 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ed7b30-477c-480e-888f-94a13fb9892d" containerName="registry" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136016 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ed7b30-477c-480e-888f-94a13fb9892d" containerName="registry" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136022 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="prom-label-proxy" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136027 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="prom-label-proxy" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136074 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="prom-label-proxy" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136081 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="alertmanager" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136086 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy-metric" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136092 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="config-reloader" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136096 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy-web" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136101 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" containerName="kube-rbac-proxy" Sep 17 01:43:28.136119 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.136106 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ed7b30-477c-480e-888f-94a13fb9892d" containerName="registry" Sep 17 01:43:28.141190 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.141173 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.143528 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.143504 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 01:43:28.143651 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.143602 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 01:43:28.143723 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.143610 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 01:43:28.144010 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.143990 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gbbn4\"" Sep 17 01:43:28.144089 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.144041 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 01:43:28.144089 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.144053 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 01:43:28.144089 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.144071 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 01:43:28.144228 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.144172 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 01:43:28.144274 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.144251 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 01:43:28.148270 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.148248 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 01:43:28.152630 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.152606 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:43:28.226433 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226390 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226433 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226434 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad3959eb-40b8-4416-aace-ab97d641a5ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226472 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad3959eb-40b8-4416-aace-ab97d641a5ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226493 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad3959eb-40b8-4416-aace-ab97d641a5ba-config-out\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226517 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ad3959eb-40b8-4416-aace-ab97d641a5ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226532 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqm57\" (UniqueName: \"kubernetes.io/projected/ad3959eb-40b8-4416-aace-ab97d641a5ba-kube-api-access-bqm57\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226551 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226569 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226618 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226643 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-web-config\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226665 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226938 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226700 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.226938 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.226749 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad3959eb-40b8-4416-aace-ab97d641a5ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.327519 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.327408 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ad3959eb-40b8-4416-aace-ab97d641a5ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.327519 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.327461 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm57\" (UniqueName: \"kubernetes.io/projected/ad3959eb-40b8-4416-aace-ab97d641a5ba-kube-api-access-bqm57\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.327779 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.327572 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.327779 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.327701 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.327779 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.327729 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.327939 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.327836 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ad3959eb-40b8-4416-aace-ab97d641a5ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.327994 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.327944 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-web-config\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.327994 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.327974 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.328095 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.328006 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.328095 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.328062 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad3959eb-40b8-4416-aace-ab97d641a5ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.328196 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.328101 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.328196 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.328132 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad3959eb-40b8-4416-aace-ab97d641a5ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.328196 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.328192 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad3959eb-40b8-4416-aace-ab97d641a5ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.328338 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.328218 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad3959eb-40b8-4416-aace-ab97d641a5ba-config-out\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.329031 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.329007 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad3959eb-40b8-4416-aace-ab97d641a5ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.329905 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.329879 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad3959eb-40b8-4416-aace-ab97d641a5ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.330630 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.330601 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.331044 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.330984 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-web-config\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.331200 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.331128 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad3959eb-40b8-4416-aace-ab97d641a5ba-config-out\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.331294 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.331262 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.331349 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.331306 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.331349 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.331263 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.332689 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.332668 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.332792 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.332774 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad3959eb-40b8-4416-aace-ab97d641a5ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.333376 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.333356 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad3959eb-40b8-4416-aace-ab97d641a5ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.335218 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.335197 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqm57\" (UniqueName: \"kubernetes.io/projected/ad3959eb-40b8-4416-aace-ab97d641a5ba-kube-api-access-bqm57\") pod \"alertmanager-main-0\" (UID: \"ad3959eb-40b8-4416-aace-ab97d641a5ba\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.451657 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.451576 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:43:28.735250 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.735216 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:43:28.739869 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:43:28.739843 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3959eb_40b8_4416_aace_ab97d641a5ba.slice/crio-dfc579b2e0a9983fb4918d86bd3c863973a667bfcf68d1c5457010f7bc04b869 WatchSource:0}: Error finding container dfc579b2e0a9983fb4918d86bd3c863973a667bfcf68d1c5457010f7bc04b869: Status 404 returned error can't find the container with id dfc579b2e0a9983fb4918d86bd3c863973a667bfcf68d1c5457010f7bc04b869 Sep 17 01:43:28.766901 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:28.766872 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aec1754e-c729-4793-b678-f8ab1274cafa" path="/var/lib/kubelet/pods/aec1754e-c729-4793-b678-f8ab1274cafa/volumes" Sep 17 01:43:29.092797 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.092687 2586 generic.go:356] "Generic (PLEG): container finished" podID="ad3959eb-40b8-4416-aace-ab97d641a5ba" containerID="b074834374dde76890701a0195c7b1a157658bd0d8b1d34dbb7a672c4e9ff9b6" exitCode=0 Sep 17 01:43:29.092797 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.092769 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad3959eb-40b8-4416-aace-ab97d641a5ba","Type":"ContainerDied","Data":"b074834374dde76890701a0195c7b1a157658bd0d8b1d34dbb7a672c4e9ff9b6"} Sep 17 01:43:29.093307 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.092884 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad3959eb-40b8-4416-aace-ab97d641a5ba","Type":"ContainerStarted","Data":"dfc579b2e0a9983fb4918d86bd3c863973a667bfcf68d1c5457010f7bc04b869"} Sep 17 01:43:29.740843 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.740793 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:43:29.743387 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.743364 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26801bb3-4b53-4677-a222-d71afd225350-metrics-certs\") pod \"network-metrics-daemon-bjxzr\" (UID: \"26801bb3-4b53-4677-a222-d71afd225350\") " pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:43:29.945522 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.945451 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt"] Sep 17 01:43:29.949277 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.949247 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:29.952048 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.952020 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 17 01:43:29.952214 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.952199 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 17 01:43:29.952325 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.952304 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 17 01:43:29.952374 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.952304 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wf8v8\"" Sep 17 01:43:29.952452 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.952431 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 17 01:43:29.952534 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.952487 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 17 01:43:29.957457 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.957435 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 17 01:43:29.964404 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.964383 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fdmc\"" Sep 17 01:43:29.968833 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.968805 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt"] Sep 17 01:43:29.973034 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:29.973006 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjxzr" Sep 17 01:43:30.044272 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.044239 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-serving-certs-ca-bundle\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.044446 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.044303 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.044446 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.044380 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.044446 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.044432 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9r6v\" (UniqueName: \"kubernetes.io/projected/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-kube-api-access-j9r6v\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.044547 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.044478 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-telemeter-client-tls\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.044547 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.044502 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-secret-telemeter-client\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.044547 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.044533 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-metrics-client-ca\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.044664 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.044554 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-federate-client-tls\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.102181 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.102149 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad3959eb-40b8-4416-aace-ab97d641a5ba","Type":"ContainerStarted","Data":"e0c4366def5fce2be69d16400ec458a18fc8aec2bf2dfbc843192a4b82b16767"} Sep 17 01:43:30.102560 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.102212 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad3959eb-40b8-4416-aace-ab97d641a5ba","Type":"ContainerStarted","Data":"f09a32f95c5ae3185bf02a43fcfc46f0cdbb6efdb80fc77ab6c8f0726aaf932b"} Sep 17 01:43:30.102560 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.102240 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad3959eb-40b8-4416-aace-ab97d641a5ba","Type":"ContainerStarted","Data":"b5151fb5ce7e77ba2e674118114a1127a166bd48bca5a6b9c0bbc5a91d34e550"} Sep 17 01:43:30.102560 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.102268 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad3959eb-40b8-4416-aace-ab97d641a5ba","Type":"ContainerStarted","Data":"76c8bda7bc46a428bdc54bb124e3c94f730be8785b076e0f0e8fe609f8b55759"} Sep 17 01:43:30.102560 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.102293 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad3959eb-40b8-4416-aace-ab97d641a5ba","Type":"ContainerStarted","Data":"5c36d7310e900262c796bb9af51a586a9e3e4ab9d5aa6b1bcbc3a391cdb8aeef"} Sep 17 01:43:30.102560 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.102317 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad3959eb-40b8-4416-aace-ab97d641a5ba","Type":"ContainerStarted","Data":"d64fc11c8b516abec1e32f80e268695afcd2d9a27c4270541d1e0f67fb248709"} Sep 17 01:43:30.128932 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.128879 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.112966744 podStartE2EDuration="2.128857839s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.891095ms" isStatefulPod=false podCreationTimestamp="2026-09-17 01:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:43:30.12756301 +0000 UTC m=+373.884330265" watchObservedRunningTime="2026-09-17 01:43:30.128857839 +0000 UTC m=+373.885625083" Sep 17 01:43:30.145644 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.145608 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-secret-telemeter-client\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.145644 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.145647 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-metrics-client-ca\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.145911 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.145664 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-federate-client-tls\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.145911 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.145818 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-serving-certs-ca-bundle\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.146031 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.145962 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.146031 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.146007 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.146136 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.146095 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9r6v\" (UniqueName: \"kubernetes.io/projected/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-kube-api-access-j9r6v\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.146209 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.146183 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-telemeter-client-tls\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.146448 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.146421 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-metrics-client-ca\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.146861 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.146835 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.147342 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.147319 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-serving-certs-ca-bundle\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.149760 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.149737 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-secret-telemeter-client\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.149876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.149820 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-federate-client-tls\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.150000 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.149982 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.150133 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.150109 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-telemeter-client-tls\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.153784 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.153760 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9r6v\" (UniqueName: \"kubernetes.io/projected/7322fa5a-a596-48a9-b0fe-3d6b76cae3ab-kube-api-access-j9r6v\") pod \"telemeter-client-6cf5cf6d8c-k68vt\" (UID: \"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab\") " pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.247970 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.247934 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bjxzr"] Sep 17 01:43:30.250988 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:43:30.250959 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26801bb3_4b53_4677_a222_d71afd225350.slice/crio-7ff54beda25e036a1c71039df0fc5c5e587336c06683cf165ebfb5fe1d194de4 WatchSource:0}: Error finding container 7ff54beda25e036a1c71039df0fc5c5e587336c06683cf165ebfb5fe1d194de4: Status 404 returned error can't find the container with id 7ff54beda25e036a1c71039df0fc5c5e587336c06683cf165ebfb5fe1d194de4 Sep 17 01:43:30.253153 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.253130 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:43:30.258980 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.258956 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" Sep 17 01:43:30.542124 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:30.542072 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt"] Sep 17 01:43:30.544959 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:43:30.544925 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7322fa5a_a596_48a9_b0fe_3d6b76cae3ab.slice/crio-c5953cdc6771b106d957fe01a4d2285fa9c6b4db3208a0c6de7429a50d3eed3c WatchSource:0}: Error finding container c5953cdc6771b106d957fe01a4d2285fa9c6b4db3208a0c6de7429a50d3eed3c: Status 404 returned error can't find the container with id c5953cdc6771b106d957fe01a4d2285fa9c6b4db3208a0c6de7429a50d3eed3c Sep 17 01:43:31.108863 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:31.108817 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" event={"ID":"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab","Type":"ContainerStarted","Data":"c5953cdc6771b106d957fe01a4d2285fa9c6b4db3208a0c6de7429a50d3eed3c"} Sep 17 01:43:31.110355 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:31.110315 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjxzr" event={"ID":"26801bb3-4b53-4677-a222-d71afd225350","Type":"ContainerStarted","Data":"7ff54beda25e036a1c71039df0fc5c5e587336c06683cf165ebfb5fe1d194de4"} Sep 17 01:43:32.117996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:32.117954 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjxzr" event={"ID":"26801bb3-4b53-4677-a222-d71afd225350","Type":"ContainerStarted","Data":"7899fa7861a506f652a6c1bf798534b12a3cb7792f49e4705265aa5d779bca0d"} Sep 17 01:43:32.118513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:32.118026 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjxzr" event={"ID":"26801bb3-4b53-4677-a222-d71afd225350","Type":"ContainerStarted","Data":"a9bc30529a9c40c8df65dff79ad13a04bfcc319c6ca651165b589872196f0964"} Sep 17 01:43:32.133879 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:32.133811 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bjxzr" podStartSLOduration=374.159987052 podStartE2EDuration="6m15.133786656s" totalImagesPullingTime="973.799604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:43:32.131400238 +0000 UTC m=+375.888167799" watchObservedRunningTime="2026-09-17 01:43:32.133786656 +0000 UTC m=+375.890553900" Sep 17 01:43:33.124411 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:33.124371 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" event={"ID":"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab","Type":"ContainerStarted","Data":"03fcb693d37d0fca52342937c2503a5b558b2ebb3cd2bd0afbb52a048263a2bf"} Sep 17 01:43:33.124820 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:33.124430 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" event={"ID":"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab","Type":"ContainerStarted","Data":"e35198d3b48db96d5b1459191f9912089f0f55e2e5f79721d3ac0fc21d0fcde2"} Sep 17 01:43:33.124820 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:33.124457 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" event={"ID":"7322fa5a-a596-48a9-b0fe-3d6b76cae3ab","Type":"ContainerStarted","Data":"d790c143304024668f2e086790307e86d54409ed99244709f6f5da2d6027f737"} Sep 17 01:43:33.145888 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:43:33.145825 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cf5cf6d8c-k68vt" podStartSLOduration=2.200492794 podStartE2EDuration="4.145807593s" totalImagesPullingTime="1.945314799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:43:33.144638237 +0000 UTC m=+376.901405493" watchObservedRunningTime="2026-09-17 01:43:33.145807593 +0000 UTC m=+376.902574836" Sep 17 01:45:50.788241 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.788207 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs"] Sep 17 01:45:50.791544 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.791527 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:50.793958 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.793937 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 01:45:50.794088 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.793942 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 01:45:50.795126 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.795107 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8wkrq\"" Sep 17 01:45:50.801261 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.801240 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs"] Sep 17 01:45:50.847471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.847432 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7r9\" (UniqueName: \"kubernetes.io/projected/12bbe316-b844-483e-bd27-db9445ae531c-kube-api-access-qp7r9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:50.847667 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.847479 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:50.847667 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.847515 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:50.948253 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.948209 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:50.948458 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.948309 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7r9\" (UniqueName: \"kubernetes.io/projected/12bbe316-b844-483e-bd27-db9445ae531c-kube-api-access-qp7r9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:50.948458 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.948345 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:50.948636 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.948615 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:50.948741 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.948724 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:50.957234 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:50.957205 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7r9\" (UniqueName: \"kubernetes.io/projected/12bbe316-b844-483e-bd27-db9445ae531c-kube-api-access-qp7r9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:51.100754 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:51.100673 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:45:51.370734 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:51.370710 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs"] Sep 17 01:45:51.373487 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:45:51.373461 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bbe316_b844_483e_bd27_db9445ae531c.slice/crio-f58ca395e1bd597979d30eea27bc8b1fc5670a93f516ea68e133e24e5fc9b2d5 WatchSource:0}: Error finding container f58ca395e1bd597979d30eea27bc8b1fc5670a93f516ea68e133e24e5fc9b2d5: Status 404 returned error can't find the container with id f58ca395e1bd597979d30eea27bc8b1fc5670a93f516ea68e133e24e5fc9b2d5 Sep 17 01:45:51.683121 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:51.683043 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" event={"ID":"12bbe316-b844-483e-bd27-db9445ae531c","Type":"ContainerStarted","Data":"f58ca395e1bd597979d30eea27bc8b1fc5670a93f516ea68e133e24e5fc9b2d5"} Sep 17 01:45:57.710571 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:57.710532 2586 generic.go:356] "Generic (PLEG): container finished" podID="12bbe316-b844-483e-bd27-db9445ae531c" containerID="03510477af3257eb82e70b8861393deb2d21a62eb40278866924ba64970694e3" exitCode=0 Sep 17 01:45:57.710970 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:45:57.710600 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" event={"ID":"12bbe316-b844-483e-bd27-db9445ae531c","Type":"ContainerDied","Data":"03510477af3257eb82e70b8861393deb2d21a62eb40278866924ba64970694e3"} Sep 17 01:46:00.725255 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:00.725219 2586 generic.go:356] "Generic (PLEG): container finished" podID="12bbe316-b844-483e-bd27-db9445ae531c" containerID="f584b352cacc7dbfaef21a989a4fbe62c10cbd3ef7db7afacee5abfc9036ccff" exitCode=0 Sep 17 01:46:00.725649 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:00.725299 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" event={"ID":"12bbe316-b844-483e-bd27-db9445ae531c","Type":"ContainerDied","Data":"f584b352cacc7dbfaef21a989a4fbe62c10cbd3ef7db7afacee5abfc9036ccff"} Sep 17 01:46:07.760818 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:07.760702 2586 generic.go:356] "Generic (PLEG): container finished" podID="12bbe316-b844-483e-bd27-db9445ae531c" containerID="4108876c5b9e3776f3391dfed733a342c76fd0f82b707dc173cb63b01daf5d10" exitCode=0 Sep 17 01:46:07.760818 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:07.760746 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" event={"ID":"12bbe316-b844-483e-bd27-db9445ae531c","Type":"ContainerDied","Data":"4108876c5b9e3776f3391dfed733a342c76fd0f82b707dc173cb63b01daf5d10"} Sep 17 01:46:08.894837 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:08.894812 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:46:09.009908 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.009868 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp7r9\" (UniqueName: \"kubernetes.io/projected/12bbe316-b844-483e-bd27-db9445ae531c-kube-api-access-qp7r9\") pod \"12bbe316-b844-483e-bd27-db9445ae531c\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " Sep 17 01:46:09.010081 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.009922 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-bundle\") pod \"12bbe316-b844-483e-bd27-db9445ae531c\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " Sep 17 01:46:09.010081 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.009967 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-util\") pod \"12bbe316-b844-483e-bd27-db9445ae531c\" (UID: \"12bbe316-b844-483e-bd27-db9445ae531c\") " Sep 17 01:46:09.010690 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.010655 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-bundle" (OuterVolumeSpecName: "bundle") pod "12bbe316-b844-483e-bd27-db9445ae531c" (UID: "12bbe316-b844-483e-bd27-db9445ae531c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:46:09.012270 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.012248 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bbe316-b844-483e-bd27-db9445ae531c-kube-api-access-qp7r9" (OuterVolumeSpecName: "kube-api-access-qp7r9") pod "12bbe316-b844-483e-bd27-db9445ae531c" (UID: "12bbe316-b844-483e-bd27-db9445ae531c"). InnerVolumeSpecName "kube-api-access-qp7r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:46:09.015206 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.015179 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-util" (OuterVolumeSpecName: "util") pod "12bbe316-b844-483e-bd27-db9445ae531c" (UID: "12bbe316-b844-483e-bd27-db9445ae531c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:46:09.111273 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.111238 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7r9\" (UniqueName: \"kubernetes.io/projected/12bbe316-b844-483e-bd27-db9445ae531c-kube-api-access-qp7r9\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:09.111273 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.111264 2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-bundle\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:09.111273 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.111273 2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bbe316-b844-483e-bd27-db9445ae531c-util\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:09.771326 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.771291 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" event={"ID":"12bbe316-b844-483e-bd27-db9445ae531c","Type":"ContainerDied","Data":"f58ca395e1bd597979d30eea27bc8b1fc5670a93f516ea68e133e24e5fc9b2d5"} Sep 17 01:46:09.771503 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.771343 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f58ca395e1bd597979d30eea27bc8b1fc5670a93f516ea68e133e24e5fc9b2d5" Sep 17 01:46:09.771503 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:09.771302 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tj4vs" Sep 17 01:46:13.628031 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.627997 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw"] Sep 17 01:46:13.628412 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.628316 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12bbe316-b844-483e-bd27-db9445ae531c" containerName="extract" Sep 17 01:46:13.628412 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.628326 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="12bbe316-b844-483e-bd27-db9445ae531c" containerName="extract" Sep 17 01:46:13.628412 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.628342 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12bbe316-b844-483e-bd27-db9445ae531c" containerName="pull" Sep 17 01:46:13.628412 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.628347 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="12bbe316-b844-483e-bd27-db9445ae531c" containerName="pull" Sep 17 01:46:13.628412 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.628356 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12bbe316-b844-483e-bd27-db9445ae531c" containerName="util" Sep 17 01:46:13.628412 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.628361 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="12bbe316-b844-483e-bd27-db9445ae531c" containerName="util" Sep 17 01:46:13.628621 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.628418 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="12bbe316-b844-483e-bd27-db9445ae531c" containerName="extract" Sep 17 01:46:13.631636 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.631618 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" Sep 17 01:46:13.635608 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.635539 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9qvsx\"" Sep 17 01:46:13.635747 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.635624 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 17 01:46:13.635747 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.635647 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:46:13.654180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.654152 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw"] Sep 17 01:46:13.748793 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.748749 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/592b6444-6b07-44b9-813f-05b99a13f95e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jcchw\" (UID: \"592b6444-6b07-44b9-813f-05b99a13f95e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" Sep 17 01:46:13.748974 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.748815 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrng\" (UniqueName: \"kubernetes.io/projected/592b6444-6b07-44b9-813f-05b99a13f95e-kube-api-access-5lrng\") pod \"cert-manager-operator-controller-manager-55c444bf86-jcchw\" (UID: \"592b6444-6b07-44b9-813f-05b99a13f95e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" Sep 17 01:46:13.849733 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.849678 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/592b6444-6b07-44b9-813f-05b99a13f95e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jcchw\" (UID: \"592b6444-6b07-44b9-813f-05b99a13f95e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" Sep 17 01:46:13.849877 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.849777 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrng\" (UniqueName: \"kubernetes.io/projected/592b6444-6b07-44b9-813f-05b99a13f95e-kube-api-access-5lrng\") pod \"cert-manager-operator-controller-manager-55c444bf86-jcchw\" (UID: \"592b6444-6b07-44b9-813f-05b99a13f95e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" Sep 17 01:46:13.850103 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.850079 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/592b6444-6b07-44b9-813f-05b99a13f95e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jcchw\" (UID: \"592b6444-6b07-44b9-813f-05b99a13f95e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" Sep 17 01:46:13.860216 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.860187 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrng\" (UniqueName: \"kubernetes.io/projected/592b6444-6b07-44b9-813f-05b99a13f95e-kube-api-access-5lrng\") pod \"cert-manager-operator-controller-manager-55c444bf86-jcchw\" (UID: \"592b6444-6b07-44b9-813f-05b99a13f95e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" Sep 17 01:46:13.940314 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:13.940229 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" Sep 17 01:46:14.216680 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:14.216547 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw"] Sep 17 01:46:14.220234 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:46:14.220208 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592b6444_6b07_44b9_813f_05b99a13f95e.slice/crio-b80e376f4f20e23ac7372a604e2462617f59492a833e1be2ed2f771993a2ab23 WatchSource:0}: Error finding container b80e376f4f20e23ac7372a604e2462617f59492a833e1be2ed2f771993a2ab23: Status 404 returned error can't find the container with id b80e376f4f20e23ac7372a604e2462617f59492a833e1be2ed2f771993a2ab23 Sep 17 01:46:14.794064 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:14.794023 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" event={"ID":"592b6444-6b07-44b9-813f-05b99a13f95e","Type":"ContainerStarted","Data":"b80e376f4f20e23ac7372a604e2462617f59492a833e1be2ed2f771993a2ab23"} Sep 17 01:46:15.800198 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:15.800160 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" event={"ID":"592b6444-6b07-44b9-813f-05b99a13f95e","Type":"ContainerStarted","Data":"0082a2d3f2d2b3147b5c96a800f12b29ccd2d3fb0d152f5ddcd1633533daf383"} Sep 17 01:46:15.824166 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:15.824116 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jcchw" podStartSLOduration=1.40344989 podStartE2EDuration="2.824098188s" totalImagesPullingTime="1.420648298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:46:15.822300874 +0000 UTC m=+539.579068130" watchObservedRunningTime="2026-09-17 01:46:15.824098188 +0000 UTC m=+539.580865432" Sep 17 01:46:17.117185 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.117150 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m"] Sep 17 01:46:17.120783 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.120765 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.123311 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.123276 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8wkrq\"" Sep 17 01:46:17.123311 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.123286 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 01:46:17.124265 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.124250 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 01:46:17.128987 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.128962 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m"] Sep 17 01:46:17.282356 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.282318 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.282543 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.282371 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzgv\" (UniqueName: \"kubernetes.io/projected/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-kube-api-access-ngzgv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.282543 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.282428 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.383436 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.383335 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzgv\" (UniqueName: \"kubernetes.io/projected/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-kube-api-access-ngzgv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.383436 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.383399 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.383652 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.383456 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.383835 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.383820 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.383873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.383853 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.391428 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.391400 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzgv\" (UniqueName: \"kubernetes.io/projected/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-kube-api-access-ngzgv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.430426 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.430391 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:17.703115 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.703080 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m"] Sep 17 01:46:17.705220 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:46:17.705192 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e89e96a_41e9_4fa7_bf43_5cd724d23b0d.slice/crio-485fc6d3e2c63be5ae224946f51a84fb133b2aecaab3f0d96a870bda82c2b373 WatchSource:0}: Error finding container 485fc6d3e2c63be5ae224946f51a84fb133b2aecaab3f0d96a870bda82c2b373: Status 404 returned error can't find the container with id 485fc6d3e2c63be5ae224946f51a84fb133b2aecaab3f0d96a870bda82c2b373 Sep 17 01:46:17.811121 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.811087 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" event={"ID":"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d","Type":"ContainerStarted","Data":"9cc79d7ba2ff7e66d2652c14d1d1ffaa0bdbef1ed7bc3ba00821c31625dd1ff9"} Sep 17 01:46:17.811245 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:17.811146 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" event={"ID":"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d","Type":"ContainerStarted","Data":"485fc6d3e2c63be5ae224946f51a84fb133b2aecaab3f0d96a870bda82c2b373"} Sep 17 01:46:18.570511 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.570473 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-trv5k"] Sep 17 01:46:18.573648 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.573631 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" Sep 17 01:46:18.575939 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.575890 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4jr6p\"" Sep 17 01:46:18.576862 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.576847 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 17 01:46:18.576960 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.576872 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 17 01:46:18.582790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.582767 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-trv5k"] Sep 17 01:46:18.697920 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.697824 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9e04a7b-5f94-42ea-aad1-9eb0f215729c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-trv5k\" (UID: \"c9e04a7b-5f94-42ea-aad1-9eb0f215729c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" Sep 17 01:46:18.697920 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.697872 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxxl\" (UniqueName: \"kubernetes.io/projected/c9e04a7b-5f94-42ea-aad1-9eb0f215729c-kube-api-access-ftxxl\") pod \"cert-manager-webhook-6486ffd997-trv5k\" (UID: \"c9e04a7b-5f94-42ea-aad1-9eb0f215729c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" Sep 17 01:46:18.799113 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.799075 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9e04a7b-5f94-42ea-aad1-9eb0f215729c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-trv5k\" (UID: \"c9e04a7b-5f94-42ea-aad1-9eb0f215729c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" Sep 17 01:46:18.799306 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.799121 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxxl\" (UniqueName: \"kubernetes.io/projected/c9e04a7b-5f94-42ea-aad1-9eb0f215729c-kube-api-access-ftxxl\") pod \"cert-manager-webhook-6486ffd997-trv5k\" (UID: \"c9e04a7b-5f94-42ea-aad1-9eb0f215729c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" Sep 17 01:46:18.807029 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.807004 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9e04a7b-5f94-42ea-aad1-9eb0f215729c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-trv5k\" (UID: \"c9e04a7b-5f94-42ea-aad1-9eb0f215729c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" Sep 17 01:46:18.807173 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.807154 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxxl\" (UniqueName: \"kubernetes.io/projected/c9e04a7b-5f94-42ea-aad1-9eb0f215729c-kube-api-access-ftxxl\") pod \"cert-manager-webhook-6486ffd997-trv5k\" (UID: \"c9e04a7b-5f94-42ea-aad1-9eb0f215729c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" Sep 17 01:46:18.818050 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.818022 2586 generic.go:356] "Generic (PLEG): container finished" podID="6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" containerID="9cc79d7ba2ff7e66d2652c14d1d1ffaa0bdbef1ed7bc3ba00821c31625dd1ff9" exitCode=0 Sep 17 01:46:18.818183 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.818112 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" event={"ID":"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d","Type":"ContainerDied","Data":"9cc79d7ba2ff7e66d2652c14d1d1ffaa0bdbef1ed7bc3ba00821c31625dd1ff9"} Sep 17 01:46:18.896263 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:18.896217 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" Sep 17 01:46:19.167312 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:19.167288 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-trv5k"] Sep 17 01:46:19.169339 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:46:19.169316 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9e04a7b_5f94_42ea_aad1_9eb0f215729c.slice/crio-420c22facf1c4484038a85bdedadc99267989d08c3a49dddc94f96baf8f18f4e WatchSource:0}: Error finding container 420c22facf1c4484038a85bdedadc99267989d08c3a49dddc94f96baf8f18f4e: Status 404 returned error can't find the container with id 420c22facf1c4484038a85bdedadc99267989d08c3a49dddc94f96baf8f18f4e Sep 17 01:46:19.823613 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:19.823556 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" event={"ID":"c9e04a7b-5f94-42ea-aad1-9eb0f215729c","Type":"ContainerStarted","Data":"420c22facf1c4484038a85bdedadc99267989d08c3a49dddc94f96baf8f18f4e"} Sep 17 01:46:20.163394 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.163349 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl"] Sep 17 01:46:20.166832 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.166810 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" Sep 17 01:46:20.169120 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.169046 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dw8fn\"" Sep 17 01:46:20.173569 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.173539 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl"] Sep 17 01:46:20.313832 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.313722 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x5v7\" (UniqueName: \"kubernetes.io/projected/38ab8992-ef89-4bf8-98b2-0df8c1ae8334-kube-api-access-8x5v7\") pod \"cert-manager-cainjector-559cdd8c87-fp5rl\" (UID: \"38ab8992-ef89-4bf8-98b2-0df8c1ae8334\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" Sep 17 01:46:20.313832 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.313803 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38ab8992-ef89-4bf8-98b2-0df8c1ae8334-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fp5rl\" (UID: \"38ab8992-ef89-4bf8-98b2-0df8c1ae8334\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" Sep 17 01:46:20.414701 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.414381 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5v7\" (UniqueName: \"kubernetes.io/projected/38ab8992-ef89-4bf8-98b2-0df8c1ae8334-kube-api-access-8x5v7\") pod \"cert-manager-cainjector-559cdd8c87-fp5rl\" (UID: \"38ab8992-ef89-4bf8-98b2-0df8c1ae8334\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" Sep 17 01:46:20.414701 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.414434 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38ab8992-ef89-4bf8-98b2-0df8c1ae8334-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fp5rl\" (UID: \"38ab8992-ef89-4bf8-98b2-0df8c1ae8334\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" Sep 17 01:46:20.423029 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.422992 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5v7\" (UniqueName: \"kubernetes.io/projected/38ab8992-ef89-4bf8-98b2-0df8c1ae8334-kube-api-access-8x5v7\") pod \"cert-manager-cainjector-559cdd8c87-fp5rl\" (UID: \"38ab8992-ef89-4bf8-98b2-0df8c1ae8334\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" Sep 17 01:46:20.423354 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.423331 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38ab8992-ef89-4bf8-98b2-0df8c1ae8334-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fp5rl\" (UID: \"38ab8992-ef89-4bf8-98b2-0df8c1ae8334\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" Sep 17 01:46:20.482054 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.481839 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" Sep 17 01:46:20.779435 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:20.779403 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl"] Sep 17 01:46:21.357193 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:46:21.357162 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38ab8992_ef89_4bf8_98b2_0df8c1ae8334.slice/crio-f150ebc8e1a113a76d03074b41309e6b75df1db68df33c43bd7cc1755a6b64f7 WatchSource:0}: Error finding container f150ebc8e1a113a76d03074b41309e6b75df1db68df33c43bd7cc1755a6b64f7: Status 404 returned error can't find the container with id f150ebc8e1a113a76d03074b41309e6b75df1db68df33c43bd7cc1755a6b64f7 Sep 17 01:46:21.834921 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:21.834843 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" event={"ID":"38ab8992-ef89-4bf8-98b2-0df8c1ae8334","Type":"ContainerStarted","Data":"f150ebc8e1a113a76d03074b41309e6b75df1db68df33c43bd7cc1755a6b64f7"} Sep 17 01:46:21.836739 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:21.836715 2586 generic.go:356] "Generic (PLEG): container finished" podID="6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" containerID="17808e01aae83a99dc6e9ca348ce98d0b076faf83411ac6a2c9ba2a923ec7dbd" exitCode=0 Sep 17 01:46:21.836827 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:21.836796 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" event={"ID":"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d","Type":"ContainerDied","Data":"17808e01aae83a99dc6e9ca348ce98d0b076faf83411ac6a2c9ba2a923ec7dbd"} Sep 17 01:46:22.846598 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:22.846526 2586 generic.go:356] "Generic (PLEG): container finished" podID="6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" containerID="0b544a3bfd80ae3c1e0826443e39204e2b07d483f83a3477024a3ec3f4d19605" exitCode=0 Sep 17 01:46:22.847085 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:22.846570 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" event={"ID":"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d","Type":"ContainerDied","Data":"0b544a3bfd80ae3c1e0826443e39204e2b07d483f83a3477024a3ec3f4d19605"} Sep 17 01:46:23.854904 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:23.854858 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" event={"ID":"c9e04a7b-5f94-42ea-aad1-9eb0f215729c","Type":"ContainerStarted","Data":"ff7d63e536ad99f5c5f3380e9f75a74860dc9fb71fd84474406d96682930a281"} Sep 17 01:46:23.855345 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:23.854932 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" Sep 17 01:46:23.856715 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:23.856689 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" event={"ID":"38ab8992-ef89-4bf8-98b2-0df8c1ae8334","Type":"ContainerStarted","Data":"df81b60d9f8a3841fe38d954b23a148e409900ebd53123b6ebf6890526b0db95"} Sep 17 01:46:23.873435 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:23.873383 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" podStartSLOduration=1.3786189819999999 podStartE2EDuration="5.873359933s" totalImagesPullingTime="4.494740951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:46:23.871447699 +0000 UTC m=+547.628214940" watchObservedRunningTime="2026-09-17 01:46:23.873359933 +0000 UTC m=+547.630127178" Sep 17 01:46:23.887932 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:23.887868 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fp5rl" podStartSLOduration=1.575126355 podStartE2EDuration="3.887842183s" totalImagesPullingTime="2.312715828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:46:23.885892864 +0000 UTC m=+547.642660129" watchObservedRunningTime="2026-09-17 01:46:23.887842183 +0000 UTC m=+547.644609428" Sep 17 01:46:23.999468 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:23.999438 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:24.148565 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.148526 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngzgv\" (UniqueName: \"kubernetes.io/projected/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-kube-api-access-ngzgv\") pod \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " Sep 17 01:46:24.148772 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.148650 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-util\") pod \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " Sep 17 01:46:24.148772 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.148722 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-bundle\") pod \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\" (UID: \"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d\") " Sep 17 01:46:24.149099 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.149075 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-bundle" (OuterVolumeSpecName: "bundle") pod "6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" (UID: "6e89e96a-41e9-4fa7-bf43-5cd724d23b0d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:46:24.150842 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.150818 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-kube-api-access-ngzgv" (OuterVolumeSpecName: "kube-api-access-ngzgv") pod "6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" (UID: "6e89e96a-41e9-4fa7-bf43-5cd724d23b0d"). InnerVolumeSpecName "kube-api-access-ngzgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:46:24.152492 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.152469 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-util" (OuterVolumeSpecName: "util") pod "6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" (UID: "6e89e96a-41e9-4fa7-bf43-5cd724d23b0d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:46:24.249894 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.249798 2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-bundle\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:24.249894 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.249833 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzgv\" (UniqueName: \"kubernetes.io/projected/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-kube-api-access-ngzgv\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:24.249894 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.249843 2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e89e96a-41e9-4fa7-bf43-5cd724d23b0d-util\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:24.863111 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.863076 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" Sep 17 01:46:24.863111 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.863103 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptx5m" event={"ID":"6e89e96a-41e9-4fa7-bf43-5cd724d23b0d","Type":"ContainerDied","Data":"485fc6d3e2c63be5ae224946f51a84fb133b2aecaab3f0d96a870bda82c2b373"} Sep 17 01:46:24.863681 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:24.863155 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="485fc6d3e2c63be5ae224946f51a84fb133b2aecaab3f0d96a870bda82c2b373" Sep 17 01:46:28.371569 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.371529 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bdfm5"] Sep 17 01:46:28.372072 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.371990 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" containerName="extract" Sep 17 01:46:28.372072 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.372009 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" containerName="extract" Sep 17 01:46:28.372072 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.372026 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" containerName="pull" Sep 17 01:46:28.372072 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.372034 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" containerName="pull" Sep 17 01:46:28.372272 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.372051 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" containerName="util" Sep 17 01:46:28.372272 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.372090 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" containerName="util" Sep 17 01:46:28.372272 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.372184 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e89e96a-41e9-4fa7-bf43-5cd724d23b0d" containerName="extract" Sep 17 01:46:28.377680 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.377659 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bdfm5" Sep 17 01:46:28.379966 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.379945 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5d2ch\"" Sep 17 01:46:28.382188 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.382049 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6967f9cb-6125-44c0-b5d7-91f8ee29ab52-bound-sa-token\") pod \"cert-manager-55b4d954d5-bdfm5\" (UID: \"6967f9cb-6125-44c0-b5d7-91f8ee29ab52\") " pod="cert-manager/cert-manager-55b4d954d5-bdfm5" Sep 17 01:46:28.382188 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.382115 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdlk\" (UniqueName: \"kubernetes.io/projected/6967f9cb-6125-44c0-b5d7-91f8ee29ab52-kube-api-access-lzdlk\") pod \"cert-manager-55b4d954d5-bdfm5\" (UID: \"6967f9cb-6125-44c0-b5d7-91f8ee29ab52\") " pod="cert-manager/cert-manager-55b4d954d5-bdfm5" Sep 17 01:46:28.384320 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.384296 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bdfm5"] Sep 17 01:46:28.482946 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.482897 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6967f9cb-6125-44c0-b5d7-91f8ee29ab52-bound-sa-token\") pod \"cert-manager-55b4d954d5-bdfm5\" (UID: \"6967f9cb-6125-44c0-b5d7-91f8ee29ab52\") " pod="cert-manager/cert-manager-55b4d954d5-bdfm5" Sep 17 01:46:28.483136 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.482956 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdlk\" (UniqueName: \"kubernetes.io/projected/6967f9cb-6125-44c0-b5d7-91f8ee29ab52-kube-api-access-lzdlk\") pod \"cert-manager-55b4d954d5-bdfm5\" (UID: \"6967f9cb-6125-44c0-b5d7-91f8ee29ab52\") " pod="cert-manager/cert-manager-55b4d954d5-bdfm5" Sep 17 01:46:28.491002 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.490969 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6967f9cb-6125-44c0-b5d7-91f8ee29ab52-bound-sa-token\") pod \"cert-manager-55b4d954d5-bdfm5\" (UID: \"6967f9cb-6125-44c0-b5d7-91f8ee29ab52\") " pod="cert-manager/cert-manager-55b4d954d5-bdfm5" Sep 17 01:46:28.491129 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.491028 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdlk\" (UniqueName: \"kubernetes.io/projected/6967f9cb-6125-44c0-b5d7-91f8ee29ab52-kube-api-access-lzdlk\") pod \"cert-manager-55b4d954d5-bdfm5\" (UID: \"6967f9cb-6125-44c0-b5d7-91f8ee29ab52\") " pod="cert-manager/cert-manager-55b4d954d5-bdfm5" Sep 17 01:46:28.687824 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.687729 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bdfm5" Sep 17 01:46:28.960566 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:28.960491 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bdfm5"] Sep 17 01:46:28.963573 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:46:28.963547 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6967f9cb_6125_44c0_b5d7_91f8ee29ab52.slice/crio-45d4ab695267523a9619e9dbdbfe5b990251663af2fd581dee72d31fcc2611e1 WatchSource:0}: Error finding container 45d4ab695267523a9619e9dbdbfe5b990251663af2fd581dee72d31fcc2611e1: Status 404 returned error can't find the container with id 45d4ab695267523a9619e9dbdbfe5b990251663af2fd581dee72d31fcc2611e1 Sep 17 01:46:29.865900 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:29.865873 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-trv5k" Sep 17 01:46:29.887138 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:29.887099 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bdfm5" event={"ID":"6967f9cb-6125-44c0-b5d7-91f8ee29ab52","Type":"ContainerStarted","Data":"555eeddef1e3450cd4aa911826165c22ce1a2e05804c9a2c4a59ede688b8ba9d"} Sep 17 01:46:29.887138 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:29.887138 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bdfm5" event={"ID":"6967f9cb-6125-44c0-b5d7-91f8ee29ab52","Type":"ContainerStarted","Data":"45d4ab695267523a9619e9dbdbfe5b990251663af2fd581dee72d31fcc2611e1"} Sep 17 01:46:29.904431 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:29.904368 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bdfm5" podStartSLOduration=1.9043526229999999 podStartE2EDuration="1.904352623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:46:29.903118127 +0000 UTC m=+553.659885396" watchObservedRunningTime="2026-09-17 01:46:29.904352623 +0000 UTC m=+553.661119867" Sep 17 01:46:37.898456 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:37.898413 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp"] Sep 17 01:46:37.964845 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:37.964807 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp"] Sep 17 01:46:37.965018 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:37.964910 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:37.967544 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:37.967518 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8wkrq\"" Sep 17 01:46:37.967544 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:37.967538 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 01:46:37.967778 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:37.967568 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 01:46:38.063970 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.063930 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2md\" (UniqueName: \"kubernetes.io/projected/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-kube-api-access-wh2md\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:38.063970 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.063970 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:38.064190 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.064068 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:38.165176 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.165080 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:38.165176 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.165143 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2md\" (UniqueName: \"kubernetes.io/projected/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-kube-api-access-wh2md\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:38.165176 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.165170 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:38.165501 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.165481 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:38.165539 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.165526 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:38.172970 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.172942 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2md\" (UniqueName: \"kubernetes.io/projected/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-kube-api-access-wh2md\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:38.274456 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.274421 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:38.568407 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.568376 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp"] Sep 17 01:46:38.570263 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:46:38.570232 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd858c9b2_13ae_4e80_b181_8ef8b932c6cf.slice/crio-6d0a55797a0ac37eb2214c4551b16b6a085c88a7535ecb4b490e64cb6caa5651 WatchSource:0}: Error finding container 6d0a55797a0ac37eb2214c4551b16b6a085c88a7535ecb4b490e64cb6caa5651: Status 404 returned error can't find the container with id 6d0a55797a0ac37eb2214c4551b16b6a085c88a7535ecb4b490e64cb6caa5651 Sep 17 01:46:38.930418 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.930382 2586 generic.go:356] "Generic (PLEG): container finished" podID="d858c9b2-13ae-4e80-b181-8ef8b932c6cf" containerID="da48c820fdd8f39d72163edb614ca9506c3b5257bb04b60cea8bcb261a76d9c0" exitCode=0 Sep 17 01:46:38.930896 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.930454 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" event={"ID":"d858c9b2-13ae-4e80-b181-8ef8b932c6cf","Type":"ContainerDied","Data":"da48c820fdd8f39d72163edb614ca9506c3b5257bb04b60cea8bcb261a76d9c0"} Sep 17 01:46:38.930896 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:38.930492 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" event={"ID":"d858c9b2-13ae-4e80-b181-8ef8b932c6cf","Type":"ContainerStarted","Data":"6d0a55797a0ac37eb2214c4551b16b6a085c88a7535ecb4b490e64cb6caa5651"} Sep 17 01:46:39.937199 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:39.937157 2586 generic.go:356] "Generic (PLEG): container finished" podID="d858c9b2-13ae-4e80-b181-8ef8b932c6cf" containerID="e6b39bd249fef2d2c8399beabba98b2d6b3d00a073cf089224c7fbdd310a8bc5" exitCode=0 Sep 17 01:46:39.937199 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:39.937201 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" event={"ID":"d858c9b2-13ae-4e80-b181-8ef8b932c6cf","Type":"ContainerDied","Data":"e6b39bd249fef2d2c8399beabba98b2d6b3d00a073cf089224c7fbdd310a8bc5"} Sep 17 01:46:40.948771 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:40.948736 2586 generic.go:356] "Generic (PLEG): container finished" podID="d858c9b2-13ae-4e80-b181-8ef8b932c6cf" containerID="e1860a49ece6095406af82894aae7129596b23e6b39bc3ccec0968e66a6c49d4" exitCode=0 Sep 17 01:46:40.949134 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:40.948815 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" event={"ID":"d858c9b2-13ae-4e80-b181-8ef8b932c6cf","Type":"ContainerDied","Data":"e1860a49ece6095406af82894aae7129596b23e6b39bc3ccec0968e66a6c49d4"} Sep 17 01:46:42.091573 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.091548 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:42.197591 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.197554 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-util\") pod \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " Sep 17 01:46:42.197760 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.197694 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh2md\" (UniqueName: \"kubernetes.io/projected/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-kube-api-access-wh2md\") pod \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " Sep 17 01:46:42.197760 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.197723 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-bundle\") pod \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\" (UID: \"d858c9b2-13ae-4e80-b181-8ef8b932c6cf\") " Sep 17 01:46:42.198478 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.198396 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-bundle" (OuterVolumeSpecName: "bundle") pod "d858c9b2-13ae-4e80-b181-8ef8b932c6cf" (UID: "d858c9b2-13ae-4e80-b181-8ef8b932c6cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:46:42.199975 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.199952 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-kube-api-access-wh2md" (OuterVolumeSpecName: "kube-api-access-wh2md") pod "d858c9b2-13ae-4e80-b181-8ef8b932c6cf" (UID: "d858c9b2-13ae-4e80-b181-8ef8b932c6cf"). InnerVolumeSpecName "kube-api-access-wh2md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:46:42.206250 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.206192 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-util" (OuterVolumeSpecName: "util") pod "d858c9b2-13ae-4e80-b181-8ef8b932c6cf" (UID: "d858c9b2-13ae-4e80-b181-8ef8b932c6cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:46:42.298624 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.298562 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2md\" (UniqueName: \"kubernetes.io/projected/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-kube-api-access-wh2md\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:42.298624 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.298619 2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-bundle\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:42.298624 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.298629 2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d858c9b2-13ae-4e80-b181-8ef8b932c6cf-util\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:42.960096 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.960064 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" Sep 17 01:46:42.960256 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.960067 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcz9fp" event={"ID":"d858c9b2-13ae-4e80-b181-8ef8b932c6cf","Type":"ContainerDied","Data":"6d0a55797a0ac37eb2214c4551b16b6a085c88a7535ecb4b490e64cb6caa5651"} Sep 17 01:46:42.960256 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:42.960189 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d0a55797a0ac37eb2214c4551b16b6a085c88a7535ecb4b490e64cb6caa5651" Sep 17 01:46:48.304368 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.304326 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn"] Sep 17 01:46:48.304897 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.304693 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d858c9b2-13ae-4e80-b181-8ef8b932c6cf" containerName="extract" Sep 17 01:46:48.304897 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.304707 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d858c9b2-13ae-4e80-b181-8ef8b932c6cf" containerName="extract" Sep 17 01:46:48.304897 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.304728 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d858c9b2-13ae-4e80-b181-8ef8b932c6cf" containerName="pull" Sep 17 01:46:48.304897 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.304736 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d858c9b2-13ae-4e80-b181-8ef8b932c6cf" containerName="pull" Sep 17 01:46:48.304897 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.304747 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d858c9b2-13ae-4e80-b181-8ef8b932c6cf" containerName="util" Sep 17 01:46:48.304897 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.304753 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d858c9b2-13ae-4e80-b181-8ef8b932c6cf" containerName="util" Sep 17 01:46:48.304897 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.304823 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d858c9b2-13ae-4e80-b181-8ef8b932c6cf" containerName="extract" Sep 17 01:46:48.307991 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.307975 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.310456 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.310429 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 01:46:48.310575 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.310463 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 01:46:48.311388 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.311368 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8wkrq\"" Sep 17 01:46:48.316808 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.316782 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn"] Sep 17 01:46:48.351621 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.351565 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.351793 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.351757 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.351836 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.351793 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ngmf\" (UniqueName: \"kubernetes.io/projected/68549d90-7f4d-4048-af54-576ea4ede69d-kube-api-access-2ngmf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.452807 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.452772 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.452807 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.452809 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ngmf\" (UniqueName: \"kubernetes.io/projected/68549d90-7f4d-4048-af54-576ea4ede69d-kube-api-access-2ngmf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.453011 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.452840 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.453173 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.453157 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.453219 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.453199 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.461921 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.461894 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ngmf\" (UniqueName: \"kubernetes.io/projected/68549d90-7f4d-4048-af54-576ea4ede69d-kube-api-access-2ngmf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.617236 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.617197 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:48.895056 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.895028 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn"] Sep 17 01:46:48.897220 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:46:48.897172 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68549d90_7f4d_4048_af54_576ea4ede69d.slice/crio-7cfe9fa79d5c3791872f022a94d92f5c9ff251320bfbe1c11f2d6e17510fc1aa WatchSource:0}: Error finding container 7cfe9fa79d5c3791872f022a94d92f5c9ff251320bfbe1c11f2d6e17510fc1aa: Status 404 returned error can't find the container with id 7cfe9fa79d5c3791872f022a94d92f5c9ff251320bfbe1c11f2d6e17510fc1aa Sep 17 01:46:48.992956 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.992912 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" event={"ID":"68549d90-7f4d-4048-af54-576ea4ede69d","Type":"ContainerStarted","Data":"af47a3322452e79d492ff3580cc28bdd0ff906ea5ffb5b20ea29951889e6501f"} Sep 17 01:46:48.993154 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:48.992965 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" event={"ID":"68549d90-7f4d-4048-af54-576ea4ede69d","Type":"ContainerStarted","Data":"7cfe9fa79d5c3791872f022a94d92f5c9ff251320bfbe1c11f2d6e17510fc1aa"} Sep 17 01:46:49.250844 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.250759 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj"] Sep 17 01:46:49.254011 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.253988 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.257023 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.256946 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 17 01:46:49.257157 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.256967 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:46:49.257157 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.256988 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 17 01:46:49.257157 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.257119 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 17 01:46:49.257316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.257169 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-djpgp\"" Sep 17 01:46:49.257316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.257173 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 17 01:46:49.265859 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.265835 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj"] Sep 17 01:46:49.358414 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.358384 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdrz\" (UniqueName: \"kubernetes.io/projected/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-kube-api-access-9sdrz\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.358414 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.358418 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-cert\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.358891 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.358485 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-manager-config\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.358891 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.358504 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-metrics-cert\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.459906 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.459866 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-manager-config\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.459906 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.459911 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-metrics-cert\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.460196 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.459941 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdrz\" (UniqueName: \"kubernetes.io/projected/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-kube-api-access-9sdrz\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.460196 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.459991 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-cert\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.460830 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.460810 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-manager-config\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.462520 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.462496 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-cert\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.462615 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.462561 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-metrics-cert\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.473236 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.473213 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdrz\" (UniqueName: \"kubernetes.io/projected/89eb875f-5994-4bd5-a74c-ed6fa36b06d9-kube-api-access-9sdrz\") pod \"lws-controller-manager-7c846775dd-x62nj\" (UID: \"89eb875f-5994-4bd5-a74c-ed6fa36b06d9\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.563484 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.563389 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:49.843754 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.843728 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj"] Sep 17 01:46:49.845655 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:46:49.845623 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89eb875f_5994_4bd5_a74c_ed6fa36b06d9.slice/crio-fcf53c41e0fee7b30c13a2eccc7db548fc57b1a13e53fe500f4fdd8cbd8fd719 WatchSource:0}: Error finding container fcf53c41e0fee7b30c13a2eccc7db548fc57b1a13e53fe500f4fdd8cbd8fd719: Status 404 returned error can't find the container with id fcf53c41e0fee7b30c13a2eccc7db548fc57b1a13e53fe500f4fdd8cbd8fd719 Sep 17 01:46:49.999840 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.999800 2586 generic.go:356] "Generic (PLEG): container finished" podID="68549d90-7f4d-4048-af54-576ea4ede69d" containerID="af47a3322452e79d492ff3580cc28bdd0ff906ea5ffb5b20ea29951889e6501f" exitCode=0 Sep 17 01:46:50.000010 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:49.999853 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" event={"ID":"68549d90-7f4d-4048-af54-576ea4ede69d","Type":"ContainerDied","Data":"af47a3322452e79d492ff3580cc28bdd0ff906ea5ffb5b20ea29951889e6501f"} Sep 17 01:46:50.001310 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.001282 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" event={"ID":"89eb875f-5994-4bd5-a74c-ed6fa36b06d9","Type":"ContainerStarted","Data":"fcf53c41e0fee7b30c13a2eccc7db548fc57b1a13e53fe500f4fdd8cbd8fd719"} Sep 17 01:46:50.443191 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.443158 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk"] Sep 17 01:46:50.447806 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.447787 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.451263 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.451242 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 17 01:46:50.451632 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.451617 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-994j7\"" Sep 17 01:46:50.451693 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.451633 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 17 01:46:50.451888 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.451862 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 17 01:46:50.452017 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.451938 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 17 01:46:50.468367 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.468324 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk"] Sep 17 01:46:50.571011 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.570975 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-manifests\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.571174 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.571025 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xxz\" (UniqueName: \"kubernetes.io/projected/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-kube-api-access-x2xxz\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.571174 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.571114 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-tmp\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.571174 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.571137 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.571174 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.571153 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.673434 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.672673 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-manifests\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.673434 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.672766 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xxz\" (UniqueName: \"kubernetes.io/projected/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-kube-api-access-x2xxz\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.673434 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.672883 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-tmp\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.673434 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.672915 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.673434 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.672946 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.675014 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.674334 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-manifests\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.678279 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.677924 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.680555 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.679519 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-tmp\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.681635 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.681608 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.686794 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.686764 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xxz\" (UniqueName: \"kubernetes.io/projected/50f81de2-ea37-4ef5-997b-c4120aeb7a0f-kube-api-access-x2xxz\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-6w7pk\" (UID: \"50f81de2-ea37-4ef5-997b-c4120aeb7a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:50.758174 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:50.758143 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:51.008391 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:51.008310 2586 generic.go:356] "Generic (PLEG): container finished" podID="68549d90-7f4d-4048-af54-576ea4ede69d" containerID="687cd90c5afde35227c89a0583d2414c47cfb6cff92ba21fac3bd251e6ce87b0" exitCode=0 Sep 17 01:46:51.008391 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:51.008350 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" event={"ID":"68549d90-7f4d-4048-af54-576ea4ede69d","Type":"ContainerDied","Data":"687cd90c5afde35227c89a0583d2414c47cfb6cff92ba21fac3bd251e6ce87b0"} Sep 17 01:46:51.036118 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:51.036089 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk"] Sep 17 01:46:51.038205 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:46:51.038178 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f81de2_ea37_4ef5_997b_c4120aeb7a0f.slice/crio-fef01e2a31287b3ceb480453dccc8f03319442d98593fc723c7fc9ca6f0a4d56 WatchSource:0}: Error finding container fef01e2a31287b3ceb480453dccc8f03319442d98593fc723c7fc9ca6f0a4d56: Status 404 returned error can't find the container with id fef01e2a31287b3ceb480453dccc8f03319442d98593fc723c7fc9ca6f0a4d56 Sep 17 01:46:52.018631 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:52.018566 2586 generic.go:356] "Generic (PLEG): container finished" podID="68549d90-7f4d-4048-af54-576ea4ede69d" containerID="1de75a771b4e75b5e92a891cca2a747da2e309a18d825aa9d3ead156b73d81a2" exitCode=0 Sep 17 01:46:52.019279 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:52.018712 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" event={"ID":"68549d90-7f4d-4048-af54-576ea4ede69d","Type":"ContainerDied","Data":"1de75a771b4e75b5e92a891cca2a747da2e309a18d825aa9d3ead156b73d81a2"} Sep 17 01:46:52.021083 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:52.021050 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" event={"ID":"50f81de2-ea37-4ef5-997b-c4120aeb7a0f","Type":"ContainerStarted","Data":"fef01e2a31287b3ceb480453dccc8f03319442d98593fc723c7fc9ca6f0a4d56"} Sep 17 01:46:53.028819 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:53.028773 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" event={"ID":"89eb875f-5994-4bd5-a74c-ed6fa36b06d9","Type":"ContainerStarted","Data":"c5a1ad80b49106b4ca5b419c73617c66bfbd9c5d9fbc429f1eba72cbc5bc7a66"} Sep 17 01:46:53.029273 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:53.029086 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:46:53.045884 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:53.045822 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" podStartSLOduration=1.734316132 podStartE2EDuration="4.045796003s" totalImagesPullingTime="2.311479871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:46:53.044207688 +0000 UTC m=+576.800974950" watchObservedRunningTime="2026-09-17 01:46:53.045796003 +0000 UTC m=+576.802563247" Sep 17 01:46:53.923850 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:53.923823 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:54.005605 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.005556 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-util\") pod \"68549d90-7f4d-4048-af54-576ea4ede69d\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " Sep 17 01:46:54.005734 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.005614 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ngmf\" (UniqueName: \"kubernetes.io/projected/68549d90-7f4d-4048-af54-576ea4ede69d-kube-api-access-2ngmf\") pod \"68549d90-7f4d-4048-af54-576ea4ede69d\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " Sep 17 01:46:54.005734 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.005690 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-bundle\") pod \"68549d90-7f4d-4048-af54-576ea4ede69d\" (UID: \"68549d90-7f4d-4048-af54-576ea4ede69d\") " Sep 17 01:46:54.006637 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.006605 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-bundle" (OuterVolumeSpecName: "bundle") pod "68549d90-7f4d-4048-af54-576ea4ede69d" (UID: "68549d90-7f4d-4048-af54-576ea4ede69d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:46:54.007934 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.007905 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68549d90-7f4d-4048-af54-576ea4ede69d-kube-api-access-2ngmf" (OuterVolumeSpecName: "kube-api-access-2ngmf") pod "68549d90-7f4d-4048-af54-576ea4ede69d" (UID: "68549d90-7f4d-4048-af54-576ea4ede69d"). InnerVolumeSpecName "kube-api-access-2ngmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:46:54.011304 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.011281 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-util" (OuterVolumeSpecName: "util") pod "68549d90-7f4d-4048-af54-576ea4ede69d" (UID: "68549d90-7f4d-4048-af54-576ea4ede69d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:46:54.036156 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.036127 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" event={"ID":"68549d90-7f4d-4048-af54-576ea4ede69d","Type":"ContainerDied","Data":"7cfe9fa79d5c3791872f022a94d92f5c9ff251320bfbe1c11f2d6e17510fc1aa"} Sep 17 01:46:54.036515 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.036171 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cfe9fa79d5c3791872f022a94d92f5c9ff251320bfbe1c11f2d6e17510fc1aa" Sep 17 01:46:54.036515 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.036255 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hk6rn" Sep 17 01:46:54.106888 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.106846 2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-bundle\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:54.106888 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.106884 2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68549d90-7f4d-4048-af54-576ea4ede69d-util\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:54.106888 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:54.106894 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ngmf\" (UniqueName: \"kubernetes.io/projected/68549d90-7f4d-4048-af54-576ea4ede69d-kube-api-access-2ngmf\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:46:55.047989 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:55.047955 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" event={"ID":"50f81de2-ea37-4ef5-997b-c4120aeb7a0f","Type":"ContainerStarted","Data":"b2ae6585edc35d8f0df372b72ef45fcc419d25b3b375574c63e2720d03431d30"} Sep 17 01:46:56.054345 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:56.054313 2586 generic.go:356] "Generic (PLEG): container finished" podID="50f81de2-ea37-4ef5-997b-c4120aeb7a0f" containerID="b2ae6585edc35d8f0df372b72ef45fcc419d25b3b375574c63e2720d03431d30" exitCode=0 Sep 17 01:46:56.054769 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:56.054361 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" event={"ID":"50f81de2-ea37-4ef5-997b-c4120aeb7a0f","Type":"ContainerDied","Data":"b2ae6585edc35d8f0df372b72ef45fcc419d25b3b375574c63e2720d03431d30"} Sep 17 01:46:57.061000 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:57.060959 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" event={"ID":"50f81de2-ea37-4ef5-997b-c4120aeb7a0f","Type":"ContainerStarted","Data":"6746ec44f890986057e17f6f223404cdb8503eb658a14c7c34c041d0568ee360"} Sep 17 01:46:57.061417 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:57.061095 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:46:57.087294 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:46:57.087239 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" podStartSLOduration=2.710772037 podStartE2EDuration="7.087221277s" totalImagesPullingTime="3.306012099s" totalInitContainerRuntime="1.070437141s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:46:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:46:57.084967394 +0000 UTC m=+580.841734650" watchObservedRunningTime="2026-09-17 01:46:57.087221277 +0000 UTC m=+580.843988521" Sep 17 01:47:04.038133 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:04.038094 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-x62nj" Sep 17 01:47:08.068045 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:08.068011 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-6w7pk" Sep 17 01:47:16.729082 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:16.729053 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:47:16.729502 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:16.729356 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:47:17.656052 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.656016 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc"] Sep 17 01:47:17.658046 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.656491 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68549d90-7f4d-4048-af54-576ea4ede69d" containerName="extract" Sep 17 01:47:17.658046 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.656513 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="68549d90-7f4d-4048-af54-576ea4ede69d" containerName="extract" Sep 17 01:47:17.658046 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.656548 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68549d90-7f4d-4048-af54-576ea4ede69d" containerName="util" Sep 17 01:47:17.658046 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.656556 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="68549d90-7f4d-4048-af54-576ea4ede69d" containerName="util" Sep 17 01:47:17.658046 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.656599 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68549d90-7f4d-4048-af54-576ea4ede69d" containerName="pull" Sep 17 01:47:17.658046 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.656608 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="68549d90-7f4d-4048-af54-576ea4ede69d" containerName="pull" Sep 17 01:47:17.658046 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.656687 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="68549d90-7f4d-4048-af54-576ea4ede69d" containerName="extract" Sep 17 01:47:17.659402 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.659385 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.662177 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.661932 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 01:47:17.662177 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.661932 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9cvvb\"" Sep 17 01:47:17.671659 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.671624 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc"] Sep 17 01:47:17.829164 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.829127 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628bh\" (UniqueName: \"kubernetes.io/projected/88a99ba0-a5cc-4003-ac5c-be586c05681d-kube-api-access-628bh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.829678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.829185 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.829678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.829220 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.829678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.829241 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.829678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.829302 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.829678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.829323 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.829678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.829405 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a99ba0-a5cc-4003-ac5c-be586c05681d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.829678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.829463 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.829678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.829545 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930257 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930166 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930257 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930204 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930459 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930261 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930459 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930302 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930459 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930344 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a99ba0-a5cc-4003-ac5c-be586c05681d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930459 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930394 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930475 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930522 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628bh\" (UniqueName: \"kubernetes.io/projected/88a99ba0-a5cc-4003-ac5c-be586c05681d-kube-api-access-628bh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930574 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930682 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930896 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930720 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.930896 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930811 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.931004 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.930893 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.931104 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.931087 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a99ba0-a5cc-4003-ac5c-be586c05681d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.932907 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.932880 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.933251 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.933229 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.942726 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.942697 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a99ba0-a5cc-4003-ac5c-be586c05681d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.943000 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.942981 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-628bh\" (UniqueName: \"kubernetes.io/projected/88a99ba0-a5cc-4003-ac5c-be586c05681d-kube-api-access-628bh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc\" (UID: \"88a99ba0-a5cc-4003-ac5c-be586c05681d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:17.973913 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:17.973871 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:18.250984 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:18.250956 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc"] Sep 17 01:47:18.253063 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:47:18.253034 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a99ba0_a5cc_4003_ac5c_be586c05681d.slice/crio-61fbf1b95adc8bb36894d944e1779feddd5e90fa0a22db5eacece118127787d9 WatchSource:0}: Error finding container 61fbf1b95adc8bb36894d944e1779feddd5e90fa0a22db5eacece118127787d9: Status 404 returned error can't find the container with id 61fbf1b95adc8bb36894d944e1779feddd5e90fa0a22db5eacece118127787d9 Sep 17 01:47:19.174284 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:19.174250 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" event={"ID":"88a99ba0-a5cc-4003-ac5c-be586c05681d","Type":"ContainerStarted","Data":"61fbf1b95adc8bb36894d944e1779feddd5e90fa0a22db5eacece118127787d9"} Sep 17 01:47:20.842921 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:20.842881 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fmtf9"] Sep 17 01:47:20.847555 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:20.847527 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:20.850936 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:20.850914 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-s25nb\"" Sep 17 01:47:20.851125 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:20.851107 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 17 01:47:20.855657 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:20.855268 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xdmm\" (UniqueName: \"kubernetes.io/projected/8eb3212a-a53c-45ed-a135-b66c4111693a-kube-api-access-5xdmm\") pod \"odh-model-controller-7998c67dc4-fmtf9\" (UID: \"8eb3212a-a53c-45ed-a135-b66c4111693a\") " pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:20.855657 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:20.855334 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8eb3212a-a53c-45ed-a135-b66c4111693a-cert\") pod \"odh-model-controller-7998c67dc4-fmtf9\" (UID: \"8eb3212a-a53c-45ed-a135-b66c4111693a\") " pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:20.860120 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:20.860098 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fmtf9"] Sep 17 01:47:20.956164 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:20.956117 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xdmm\" (UniqueName: \"kubernetes.io/projected/8eb3212a-a53c-45ed-a135-b66c4111693a-kube-api-access-5xdmm\") pod \"odh-model-controller-7998c67dc4-fmtf9\" (UID: \"8eb3212a-a53c-45ed-a135-b66c4111693a\") " pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:20.956360 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:20.956196 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8eb3212a-a53c-45ed-a135-b66c4111693a-cert\") pod \"odh-model-controller-7998c67dc4-fmtf9\" (UID: \"8eb3212a-a53c-45ed-a135-b66c4111693a\") " pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:20.956360 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:47:20.956334 2586 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 17 01:47:20.956450 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:47:20.956409 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eb3212a-a53c-45ed-a135-b66c4111693a-cert podName:8eb3212a-a53c-45ed-a135-b66c4111693a nodeName:}" failed. No retries permitted until 2026-09-17 01:47:21.456392409 +0000 UTC m=+605.213159631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8eb3212a-a53c-45ed-a135-b66c4111693a-cert") pod "odh-model-controller-7998c67dc4-fmtf9" (UID: "8eb3212a-a53c-45ed-a135-b66c4111693a") : secret "odh-model-controller-webhook-cert" not found Sep 17 01:47:20.967100 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:20.967058 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xdmm\" (UniqueName: \"kubernetes.io/projected/8eb3212a-a53c-45ed-a135-b66c4111693a-kube-api-access-5xdmm\") pod \"odh-model-controller-7998c67dc4-fmtf9\" (UID: \"8eb3212a-a53c-45ed-a135-b66c4111693a\") " pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:21.394938 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.394899 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7c5n9"] Sep 17 01:47:21.398327 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.398309 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:21.400842 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.400818 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-8b9wr\"" Sep 17 01:47:21.400969 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.400817 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 17 01:47:21.409648 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.409620 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7c5n9"] Sep 17 01:47:21.460012 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.459973 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xv6\" (UniqueName: \"kubernetes.io/projected/c89a9803-f008-4a91-820a-926e14f3bfcb-kube-api-access-t2xv6\") pod \"model-serving-api-54d89bd5fc-7c5n9\" (UID: \"c89a9803-f008-4a91-820a-926e14f3bfcb\") " pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:21.460220 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.460037 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8eb3212a-a53c-45ed-a135-b66c4111693a-cert\") pod \"odh-model-controller-7998c67dc4-fmtf9\" (UID: \"8eb3212a-a53c-45ed-a135-b66c4111693a\") " pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:21.460220 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.460072 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89a9803-f008-4a91-820a-926e14f3bfcb-tls-certs\") pod \"model-serving-api-54d89bd5fc-7c5n9\" (UID: \"c89a9803-f008-4a91-820a-926e14f3bfcb\") " pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:21.462644 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.462621 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8eb3212a-a53c-45ed-a135-b66c4111693a-cert\") pod \"odh-model-controller-7998c67dc4-fmtf9\" (UID: \"8eb3212a-a53c-45ed-a135-b66c4111693a\") " pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:21.560884 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.560826 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xv6\" (UniqueName: \"kubernetes.io/projected/c89a9803-f008-4a91-820a-926e14f3bfcb-kube-api-access-t2xv6\") pod \"model-serving-api-54d89bd5fc-7c5n9\" (UID: \"c89a9803-f008-4a91-820a-926e14f3bfcb\") " pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:21.561090 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.560913 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89a9803-f008-4a91-820a-926e14f3bfcb-tls-certs\") pod \"model-serving-api-54d89bd5fc-7c5n9\" (UID: \"c89a9803-f008-4a91-820a-926e14f3bfcb\") " pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:21.565607 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:47:21.561538 2586 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 17 01:47:21.565607 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:47:21.561643 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89a9803-f008-4a91-820a-926e14f3bfcb-tls-certs podName:c89a9803-f008-4a91-820a-926e14f3bfcb nodeName:}" failed. No retries permitted until 2026-09-17 01:47:22.061621546 +0000 UTC m=+605.818388784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c89a9803-f008-4a91-820a-926e14f3bfcb-tls-certs") pod "model-serving-api-54d89bd5fc-7c5n9" (UID: "c89a9803-f008-4a91-820a-926e14f3bfcb") : secret "model-serving-api-tls" not found Sep 17 01:47:21.576313 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.576278 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xv6\" (UniqueName: \"kubernetes.io/projected/c89a9803-f008-4a91-820a-926e14f3bfcb-kube-api-access-t2xv6\") pod \"model-serving-api-54d89bd5fc-7c5n9\" (UID: \"c89a9803-f008-4a91-820a-926e14f3bfcb\") " pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:21.760412 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:21.760311 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:22.031876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:22.031724 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fmtf9"] Sep 17 01:47:22.034502 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:47:22.034469 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb3212a_a53c_45ed_a135_b66c4111693a.slice/crio-b3bf58a1c0228c18257cdf4d7f90722e967c3e15d5b1b96ce02074787e4a7e17 WatchSource:0}: Error finding container b3bf58a1c0228c18257cdf4d7f90722e967c3e15d5b1b96ce02074787e4a7e17: Status 404 returned error can't find the container with id b3bf58a1c0228c18257cdf4d7f90722e967c3e15d5b1b96ce02074787e4a7e17 Sep 17 01:47:22.066292 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:22.066246 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89a9803-f008-4a91-820a-926e14f3bfcb-tls-certs\") pod \"model-serving-api-54d89bd5fc-7c5n9\" (UID: \"c89a9803-f008-4a91-820a-926e14f3bfcb\") " pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:22.069034 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:22.069004 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89a9803-f008-4a91-820a-926e14f3bfcb-tls-certs\") pod \"model-serving-api-54d89bd5fc-7c5n9\" (UID: \"c89a9803-f008-4a91-820a-926e14f3bfcb\") " pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:22.190503 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:22.190464 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" event={"ID":"8eb3212a-a53c-45ed-a135-b66c4111693a","Type":"ContainerStarted","Data":"b3bf58a1c0228c18257cdf4d7f90722e967c3e15d5b1b96ce02074787e4a7e17"} Sep 17 01:47:22.309924 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:22.309832 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:22.582333 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:22.582234 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7c5n9"] Sep 17 01:47:22.585513 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:47:22.585484 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89a9803_f008_4a91_820a_926e14f3bfcb.slice/crio-a4275a44810158df7e8fc09de22c7d15831b0e14db64ca3b01cd887735c32366 WatchSource:0}: Error finding container a4275a44810158df7e8fc09de22c7d15831b0e14db64ca3b01cd887735c32366: Status 404 returned error can't find the container with id a4275a44810158df7e8fc09de22c7d15831b0e14db64ca3b01cd887735c32366 Sep 17 01:47:23.203794 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:23.203706 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" event={"ID":"c89a9803-f008-4a91-820a-926e14f3bfcb","Type":"ContainerStarted","Data":"a4275a44810158df7e8fc09de22c7d15831b0e14db64ca3b01cd887735c32366"} Sep 17 01:47:26.221295 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:26.221262 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" event={"ID":"8eb3212a-a53c-45ed-a135-b66c4111693a","Type":"ContainerStarted","Data":"695af3e583fe03b6b2167672370d32580fc848f2d982c61b135f283a3de53671"} Sep 17 01:47:26.221802 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:26.221389 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:26.223213 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:26.223189 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" event={"ID":"c89a9803-f008-4a91-820a-926e14f3bfcb","Type":"ContainerStarted","Data":"17355fe4086815ce19794e7b8b093e7e8c905bce9aa2302f708dae29e5f64d30"} Sep 17 01:47:26.223330 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:26.223243 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:26.239259 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:26.239211 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" podStartSLOduration=2.256856728 podStartE2EDuration="6.239195214s" totalImagesPullingTime="3.982338486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:47:26.237262786 +0000 UTC m=+609.994030078" watchObservedRunningTime="2026-09-17 01:47:26.239195214 +0000 UTC m=+609.995962469" Sep 17 01:47:26.252888 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:26.252842 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" podStartSLOduration=1.778922554 podStartE2EDuration="5.252827741s" totalImagesPullingTime="3.473905187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:47:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:47:26.251912713 +0000 UTC m=+610.008679979" watchObservedRunningTime="2026-09-17 01:47:26.252827741 +0000 UTC m=+610.009594984" Sep 17 01:47:29.917488 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:29.917450 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d"] Sep 17 01:47:29.921233 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:29.921213 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:47:29.924307 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:29.924286 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 17 01:47:29.924568 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:29.924551 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mrlzd\"" Sep 17 01:47:29.929039 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:29.929003 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g"] Sep 17 01:47:29.932926 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:29.932907 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d"] Sep 17 01:47:29.933035 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:29.933009 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" Sep 17 01:47:29.935421 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:29.935399 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 17 01:47:29.935524 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:29.935425 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2nxd5\"" Sep 17 01:47:29.943707 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:29.943683 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g"] Sep 17 01:47:30.043338 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.043300 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5147556f-49a9-46a5-80bb-49c1009cd777-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdl2d\" (UID: \"5147556f-49a9-46a5-80bb-49c1009cd777\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:47:30.043514 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.043345 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10322d15-0304-4495-88a9-93e82970c720-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dqr6g\" (UID: \"10322d15-0304-4495-88a9-93e82970c720\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" Sep 17 01:47:30.043514 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.043468 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrdq\" (UniqueName: \"kubernetes.io/projected/10322d15-0304-4495-88a9-93e82970c720-kube-api-access-gvrdq\") pod \"llmisvc-controller-manager-777c6bd7d5-dqr6g\" (UID: \"10322d15-0304-4495-88a9-93e82970c720\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" Sep 17 01:47:30.043514 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.043505 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcd8\" (UniqueName: \"kubernetes.io/projected/5147556f-49a9-46a5-80bb-49c1009cd777-kube-api-access-hqcd8\") pod \"kserve-controller-manager-d6f4b65b8-xdl2d\" (UID: \"5147556f-49a9-46a5-80bb-49c1009cd777\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:47:30.144668 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.144627 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrdq\" (UniqueName: \"kubernetes.io/projected/10322d15-0304-4495-88a9-93e82970c720-kube-api-access-gvrdq\") pod \"llmisvc-controller-manager-777c6bd7d5-dqr6g\" (UID: \"10322d15-0304-4495-88a9-93e82970c720\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" Sep 17 01:47:30.144668 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.144670 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcd8\" (UniqueName: \"kubernetes.io/projected/5147556f-49a9-46a5-80bb-49c1009cd777-kube-api-access-hqcd8\") pod \"kserve-controller-manager-d6f4b65b8-xdl2d\" (UID: \"5147556f-49a9-46a5-80bb-49c1009cd777\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:47:30.144932 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.144723 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5147556f-49a9-46a5-80bb-49c1009cd777-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdl2d\" (UID: \"5147556f-49a9-46a5-80bb-49c1009cd777\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:47:30.144932 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.144743 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10322d15-0304-4495-88a9-93e82970c720-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dqr6g\" (UID: \"10322d15-0304-4495-88a9-93e82970c720\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" Sep 17 01:47:30.144932 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:47:30.144859 2586 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 17 01:47:30.144932 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:47:30.144922 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5147556f-49a9-46a5-80bb-49c1009cd777-cert podName:5147556f-49a9-46a5-80bb-49c1009cd777 nodeName:}" failed. No retries permitted until 2026-09-17 01:47:30.644904214 +0000 UTC m=+614.401671450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5147556f-49a9-46a5-80bb-49c1009cd777-cert") pod "kserve-controller-manager-d6f4b65b8-xdl2d" (UID: "5147556f-49a9-46a5-80bb-49c1009cd777") : secret "kserve-webhook-server-cert" not found Sep 17 01:47:30.147323 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.147306 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10322d15-0304-4495-88a9-93e82970c720-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dqr6g\" (UID: \"10322d15-0304-4495-88a9-93e82970c720\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" Sep 17 01:47:30.153786 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.153761 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrdq\" (UniqueName: \"kubernetes.io/projected/10322d15-0304-4495-88a9-93e82970c720-kube-api-access-gvrdq\") pod \"llmisvc-controller-manager-777c6bd7d5-dqr6g\" (UID: \"10322d15-0304-4495-88a9-93e82970c720\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" Sep 17 01:47:30.154159 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.154137 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcd8\" (UniqueName: \"kubernetes.io/projected/5147556f-49a9-46a5-80bb-49c1009cd777-kube-api-access-hqcd8\") pod \"kserve-controller-manager-d6f4b65b8-xdl2d\" (UID: \"5147556f-49a9-46a5-80bb-49c1009cd777\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:47:30.245498 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.245403 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" Sep 17 01:47:30.529662 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.529636 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g"] Sep 17 01:47:30.531765 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:47:30.531738 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod10322d15_0304_4495_88a9_93e82970c720.slice/crio-12d2307d76a3a3890ad08378a0bd3383a4b3cd32bc0a8410cb30a0c967c7ecab WatchSource:0}: Error finding container 12d2307d76a3a3890ad08378a0bd3383a4b3cd32bc0a8410cb30a0c967c7ecab: Status 404 returned error can't find the container with id 12d2307d76a3a3890ad08378a0bd3383a4b3cd32bc0a8410cb30a0c967c7ecab Sep 17 01:47:30.649659 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.649623 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5147556f-49a9-46a5-80bb-49c1009cd777-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdl2d\" (UID: \"5147556f-49a9-46a5-80bb-49c1009cd777\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:47:30.652236 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.652207 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5147556f-49a9-46a5-80bb-49c1009cd777-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdl2d\" (UID: \"5147556f-49a9-46a5-80bb-49c1009cd777\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:47:30.834400 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:30.834314 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:47:31.122324 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:31.122295 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d"] Sep 17 01:47:31.124197 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:47:31.124166 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5147556f_49a9_46a5_80bb_49c1009cd777.slice/crio-197d6fa14c711d981e3b3b735d779d4821092844f0e9f86de70cec01a0c2b72d WatchSource:0}: Error finding container 197d6fa14c711d981e3b3b735d779d4821092844f0e9f86de70cec01a0c2b72d: Status 404 returned error can't find the container with id 197d6fa14c711d981e3b3b735d779d4821092844f0e9f86de70cec01a0c2b72d Sep 17 01:47:31.255102 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:31.255043 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" event={"ID":"5147556f-49a9-46a5-80bb-49c1009cd777","Type":"ContainerStarted","Data":"197d6fa14c711d981e3b3b735d779d4821092844f0e9f86de70cec01a0c2b72d"} Sep 17 01:47:31.256511 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:31.256477 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" event={"ID":"10322d15-0304-4495-88a9-93e82970c720","Type":"ContainerStarted","Data":"12d2307d76a3a3890ad08378a0bd3383a4b3cd32bc0a8410cb30a0c967c7ecab"} Sep 17 01:47:34.280125 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:34.280089 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" event={"ID":"5147556f-49a9-46a5-80bb-49c1009cd777","Type":"ContainerStarted","Data":"2c1c7e1bf2edce5e72dff563c62a289811437bf1e7ebaadde92bbb3e9ca920de"} Sep 17 01:47:34.280571 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:34.280254 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:47:34.281842 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:34.281819 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" event={"ID":"10322d15-0304-4495-88a9-93e82970c720","Type":"ContainerStarted","Data":"ebd07755ec390383e05040bf40eff71192ded058db0bf5513cd16fc1bdff1751"} Sep 17 01:47:34.281947 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:34.281865 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" Sep 17 01:47:34.298053 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:34.298004 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" podStartSLOduration=2.818554659 podStartE2EDuration="5.297986971s" totalImagesPullingTime="2.479432312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:47:34.296828518 +0000 UTC m=+618.053595784" watchObservedRunningTime="2026-09-17 01:47:34.297986971 +0000 UTC m=+618.054754228" Sep 17 01:47:34.330134 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:34.330079 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" podStartSLOduration=2.260299936 podStartE2EDuration="5.330061927s" totalImagesPullingTime="3.069761991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:47:34.329844934 +0000 UTC m=+618.086612215" watchObservedRunningTime="2026-09-17 01:47:34.330061927 +0000 UTC m=+618.086829171" Sep 17 01:47:37.231396 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:37.231351 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fmtf9" Sep 17 01:47:37.233943 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:37.233921 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-7c5n9" Sep 17 01:47:45.416151 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:45.416107 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:47:45.416407 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:45.416190 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:47:45.416407 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:45.416223 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:47:46.352169 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:46.352126 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" event={"ID":"88a99ba0-a5cc-4003-ac5c-be586c05681d","Type":"ContainerStarted","Data":"671166d0392f563b08c80fd429ab05b9db2d38cc95d37ae73e2eff404c704c11"} Sep 17 01:47:46.377400 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:46.377336 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" podStartSLOduration=2.216756034 podStartE2EDuration="29.377312847s" totalImagesPullingTime="27.160556813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:47:46.376302202 +0000 UTC m=+630.133069466" watchObservedRunningTime="2026-09-17 01:47:46.377312847 +0000 UTC m=+630.134080091" Sep 17 01:47:46.974337 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:46.974297 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:46.979349 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:46.979321 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:47.357601 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:47.357562 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:47:47.358555 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:47:47.358535 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc" Sep 17 01:48:05.289202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:05.289168 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dqr6g" Sep 17 01:48:05.292295 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:05.292276 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdl2d" Sep 17 01:48:25.539252 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.539220 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl"] Sep 17 01:48:25.549546 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.549520 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.550005 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.549979 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl"] Sep 17 01:48:25.552045 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.552021 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 01:48:25.552173 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.552089 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 01:48:25.552173 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.552091 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8wkrq\"" Sep 17 01:48:25.645906 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.645870 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj"] Sep 17 01:48:25.652439 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.651704 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.657052 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.657024 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj"] Sep 17 01:48:25.660257 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.660235 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8b42\" (UniqueName: \"kubernetes.io/projected/817ba496-7206-4225-aefc-f90ba58284d3-kube-api-access-g8b42\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.660353 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.660284 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.660400 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.660352 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.742331 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.742294 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs"] Sep 17 01:48:25.746440 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.746423 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:25.754142 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.754117 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs"] Sep 17 01:48:25.761874 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.761849 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8b42\" (UniqueName: \"kubernetes.io/projected/817ba496-7206-4225-aefc-f90ba58284d3-kube-api-access-g8b42\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.761997 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.761886 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.761997 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.761910 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.761997 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.761937 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.762726 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.762020 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.762726 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.762058 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w22p2\" (UniqueName: \"kubernetes.io/projected/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-kube-api-access-w22p2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.762726 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.762302 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.762726 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.762335 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.769897 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.769873 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8b42\" (UniqueName: \"kubernetes.io/projected/817ba496-7206-4225-aefc-f90ba58284d3-kube-api-access-g8b42\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.843479 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.843379 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx"] Sep 17 01:48:25.847451 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.847434 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:25.855326 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.855293 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx"] Sep 17 01:48:25.860201 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.860174 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:25.863791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.863241 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.863791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.863285 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.863791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.863322 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:25.863791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.863438 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:25.863791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.863472 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w22p2\" (UniqueName: \"kubernetes.io/projected/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-kube-api-access-w22p2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.863791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.863781 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.864198 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.863983 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkwzz\" (UniqueName: \"kubernetes.io/projected/b296384c-2609-4ce0-9a98-461cddd3b0cb-kube-api-access-jkwzz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:25.864517 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.864475 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.874194 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.873855 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w22p2\" (UniqueName: \"kubernetes.io/projected/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-kube-api-access-w22p2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.963118 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.963080 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:25.965526 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.965503 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:25.965653 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.965541 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hnzj\" (UniqueName: \"kubernetes.io/projected/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-kube-api-access-7hnzj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:25.965653 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.965601 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwzz\" (UniqueName: \"kubernetes.io/projected/b296384c-2609-4ce0-9a98-461cddd3b0cb-kube-api-access-jkwzz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:25.965653 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.965637 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:25.965771 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.965746 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:25.965812 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.965804 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:25.965935 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.965914 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:25.966083 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.966066 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:25.977543 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:25.977516 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwzz\" (UniqueName: \"kubernetes.io/projected/b296384c-2609-4ce0-9a98-461cddd3b0cb-kube-api-access-jkwzz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:26.055923 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.055884 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:26.066974 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.066939 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnzj\" (UniqueName: \"kubernetes.io/projected/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-kube-api-access-7hnzj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:26.067123 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.067028 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:26.067123 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.067057 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:26.067432 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.067394 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:26.067504 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.067483 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:26.075530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.075501 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnzj\" (UniqueName: \"kubernetes.io/projected/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-kube-api-access-7hnzj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:26.143425 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:48:26.143378 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817ba496_7206_4225_aefc_f90ba58284d3.slice/crio-55acd4150730ad7aaf7e40f03031360695e51f6b66b13162cfa952526268e203 WatchSource:0}: Error finding container 55acd4150730ad7aaf7e40f03031360695e51f6b66b13162cfa952526268e203: Status 404 returned error can't find the container with id 55acd4150730ad7aaf7e40f03031360695e51f6b66b13162cfa952526268e203 Sep 17 01:48:26.143536 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.143482 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl"] Sep 17 01:48:26.157965 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.157942 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:26.245496 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:48:26.245466 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8000fe1_9b79_4d1e_a74a_4fc9fdb960de.slice/crio-1b352b3586ad316e7e64948b688e8d0209da3eb2403556383e27c2b81b65c2ed WatchSource:0}: Error finding container 1b352b3586ad316e7e64948b688e8d0209da3eb2403556383e27c2b81b65c2ed: Status 404 returned error can't find the container with id 1b352b3586ad316e7e64948b688e8d0209da3eb2403556383e27c2b81b65c2ed Sep 17 01:48:26.245819 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.245789 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj"] Sep 17 01:48:26.335017 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.334987 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs"] Sep 17 01:48:26.336387 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:48:26.336365 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb296384c_2609_4ce0_9a98_461cddd3b0cb.slice/crio-1d6c3b3ef7263988e19adc87190e2674911b2b5cd02b4287742d8287f7dd9f3c WatchSource:0}: Error finding container 1d6c3b3ef7263988e19adc87190e2674911b2b5cd02b4287742d8287f7dd9f3c: Status 404 returned error can't find the container with id 1d6c3b3ef7263988e19adc87190e2674911b2b5cd02b4287742d8287f7dd9f3c Sep 17 01:48:26.429812 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.429784 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx"] Sep 17 01:48:26.444559 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:48:26.444529 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d07e4b_9239_4464_8bd3_6707c7ad51c5.slice/crio-f9d652b9db25e5652bf364e4584f602ac33b3eb96f6ff0292561f1a7aac15658 WatchSource:0}: Error finding container f9d652b9db25e5652bf364e4584f602ac33b3eb96f6ff0292561f1a7aac15658: Status 404 returned error can't find the container with id f9d652b9db25e5652bf364e4584f602ac33b3eb96f6ff0292561f1a7aac15658 Sep 17 01:48:26.569141 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.569108 2586 generic.go:356] "Generic (PLEG): container finished" podID="d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" containerID="2cf617007c29cad01ef6b285a3728e5011d142ef0439059b024aa71f02e869b7" exitCode=0 Sep 17 01:48:26.569614 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.569213 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" event={"ID":"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de","Type":"ContainerDied","Data":"2cf617007c29cad01ef6b285a3728e5011d142ef0439059b024aa71f02e869b7"} Sep 17 01:48:26.569614 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.569283 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" event={"ID":"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de","Type":"ContainerStarted","Data":"1b352b3586ad316e7e64948b688e8d0209da3eb2403556383e27c2b81b65c2ed"} Sep 17 01:48:26.571123 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.571097 2586 generic.go:356] "Generic (PLEG): container finished" podID="b296384c-2609-4ce0-9a98-461cddd3b0cb" containerID="7da0bb70210cb1f9cfc9e6d42ef1a543535fe7258e2c7030995478e9f157a093" exitCode=0 Sep 17 01:48:26.571235 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.571183 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" event={"ID":"b296384c-2609-4ce0-9a98-461cddd3b0cb","Type":"ContainerDied","Data":"7da0bb70210cb1f9cfc9e6d42ef1a543535fe7258e2c7030995478e9f157a093"} Sep 17 01:48:26.571290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.571233 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" event={"ID":"b296384c-2609-4ce0-9a98-461cddd3b0cb","Type":"ContainerStarted","Data":"1d6c3b3ef7263988e19adc87190e2674911b2b5cd02b4287742d8287f7dd9f3c"} Sep 17 01:48:26.573094 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.573070 2586 generic.go:356] "Generic (PLEG): container finished" podID="817ba496-7206-4225-aefc-f90ba58284d3" containerID="48a72067b11a2fca70c0a733696ecfde951428aa32c8602f372172f7a686d768" exitCode=0 Sep 17 01:48:26.573184 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.573155 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" event={"ID":"817ba496-7206-4225-aefc-f90ba58284d3","Type":"ContainerDied","Data":"48a72067b11a2fca70c0a733696ecfde951428aa32c8602f372172f7a686d768"} Sep 17 01:48:26.573239 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.573199 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" event={"ID":"817ba496-7206-4225-aefc-f90ba58284d3","Type":"ContainerStarted","Data":"55acd4150730ad7aaf7e40f03031360695e51f6b66b13162cfa952526268e203"} Sep 17 01:48:26.574974 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.574955 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" event={"ID":"a1d07e4b-9239-4464-8bd3-6707c7ad51c5","Type":"ContainerStarted","Data":"1c68b98877edc354b7e746796137bc2060db3bbd1943d35a2ac072ddf06e7aca"} Sep 17 01:48:26.575087 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:26.574989 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" event={"ID":"a1d07e4b-9239-4464-8bd3-6707c7ad51c5","Type":"ContainerStarted","Data":"f9d652b9db25e5652bf364e4584f602ac33b3eb96f6ff0292561f1a7aac15658"} Sep 17 01:48:27.591039 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:27.591005 2586 generic.go:356] "Generic (PLEG): container finished" podID="a1d07e4b-9239-4464-8bd3-6707c7ad51c5" containerID="1c68b98877edc354b7e746796137bc2060db3bbd1943d35a2ac072ddf06e7aca" exitCode=0 Sep 17 01:48:27.591484 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:27.591098 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" event={"ID":"a1d07e4b-9239-4464-8bd3-6707c7ad51c5","Type":"ContainerDied","Data":"1c68b98877edc354b7e746796137bc2060db3bbd1943d35a2ac072ddf06e7aca"} Sep 17 01:48:27.595109 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:27.595050 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" event={"ID":"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de","Type":"ContainerStarted","Data":"206cd5b3ce3b9b10dcb4e216dc7f98a1d3f7a24be9986af15d31de7060f91d0a"} Sep 17 01:48:27.597510 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:27.597486 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" event={"ID":"817ba496-7206-4225-aefc-f90ba58284d3","Type":"ContainerStarted","Data":"6e22120de7f9b8e71af87177cc47c2412b5b259f438aa111a411434342c6a034"} Sep 17 01:48:28.605350 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:28.605321 2586 generic.go:356] "Generic (PLEG): container finished" podID="d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" containerID="206cd5b3ce3b9b10dcb4e216dc7f98a1d3f7a24be9986af15d31de7060f91d0a" exitCode=0 Sep 17 01:48:28.605761 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:28.605401 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" event={"ID":"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de","Type":"ContainerDied","Data":"206cd5b3ce3b9b10dcb4e216dc7f98a1d3f7a24be9986af15d31de7060f91d0a"} Sep 17 01:48:28.607331 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:28.607308 2586 generic.go:356] "Generic (PLEG): container finished" podID="b296384c-2609-4ce0-9a98-461cddd3b0cb" containerID="e19d2386f5c6a1be7fc6f2d05c51f4af52fbe9c0533ff28528b466e412c668e4" exitCode=0 Sep 17 01:48:28.607468 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:28.607406 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" event={"ID":"b296384c-2609-4ce0-9a98-461cddd3b0cb","Type":"ContainerDied","Data":"e19d2386f5c6a1be7fc6f2d05c51f4af52fbe9c0533ff28528b466e412c668e4"} Sep 17 01:48:28.609448 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:28.609430 2586 generic.go:356] "Generic (PLEG): container finished" podID="817ba496-7206-4225-aefc-f90ba58284d3" containerID="6e22120de7f9b8e71af87177cc47c2412b5b259f438aa111a411434342c6a034" exitCode=0 Sep 17 01:48:28.609573 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:28.609553 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" event={"ID":"817ba496-7206-4225-aefc-f90ba58284d3","Type":"ContainerDied","Data":"6e22120de7f9b8e71af87177cc47c2412b5b259f438aa111a411434342c6a034"} Sep 17 01:48:28.611591 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:28.611559 2586 generic.go:356] "Generic (PLEG): container finished" podID="a1d07e4b-9239-4464-8bd3-6707c7ad51c5" containerID="e293556281fa7b1f1fb253995c446a2010215ab7b14aa4e3fd142346a67353be" exitCode=0 Sep 17 01:48:28.611677 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:28.611621 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" event={"ID":"a1d07e4b-9239-4464-8bd3-6707c7ad51c5","Type":"ContainerDied","Data":"e293556281fa7b1f1fb253995c446a2010215ab7b14aa4e3fd142346a67353be"} Sep 17 01:48:29.619221 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:29.619159 2586 generic.go:356] "Generic (PLEG): container finished" podID="b296384c-2609-4ce0-9a98-461cddd3b0cb" containerID="be49a3cbc9421da27b460850a635e4a914c26c50a1eb8fa4aaa06fa9a2b5bcd1" exitCode=0 Sep 17 01:48:29.619676 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:29.619249 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" event={"ID":"b296384c-2609-4ce0-9a98-461cddd3b0cb","Type":"ContainerDied","Data":"be49a3cbc9421da27b460850a635e4a914c26c50a1eb8fa4aaa06fa9a2b5bcd1"} Sep 17 01:48:29.621478 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:29.621451 2586 generic.go:356] "Generic (PLEG): container finished" podID="817ba496-7206-4225-aefc-f90ba58284d3" containerID="0183226cdc27587e7016a124db62a4657a920d19c08b48ec93d002cea5752a3a" exitCode=0 Sep 17 01:48:29.621622 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:29.621508 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" event={"ID":"817ba496-7206-4225-aefc-f90ba58284d3","Type":"ContainerDied","Data":"0183226cdc27587e7016a124db62a4657a920d19c08b48ec93d002cea5752a3a"} Sep 17 01:48:29.623782 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:29.623761 2586 generic.go:356] "Generic (PLEG): container finished" podID="a1d07e4b-9239-4464-8bd3-6707c7ad51c5" containerID="17c117abb8ff61c1594e3a4978fd26293321958d516d076830bd1c3a6fc4e077" exitCode=0 Sep 17 01:48:29.623897 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:29.623783 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" event={"ID":"a1d07e4b-9239-4464-8bd3-6707c7ad51c5","Type":"ContainerDied","Data":"17c117abb8ff61c1594e3a4978fd26293321958d516d076830bd1c3a6fc4e077"} Sep 17 01:48:29.625900 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:29.625880 2586 generic.go:356] "Generic (PLEG): container finished" podID="d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" containerID="64edb8b8822d894e3afaeef8285961cc79e67fd15eb754f816cb338e5aeab1f2" exitCode=0 Sep 17 01:48:29.625977 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:29.625906 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" event={"ID":"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de","Type":"ContainerDied","Data":"64edb8b8822d894e3afaeef8285961cc79e67fd15eb754f816cb338e5aeab1f2"} Sep 17 01:48:30.778657 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.778628 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:48:30.820549 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.820521 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:30.845941 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.845911 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:30.849304 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.849282 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:30.909944 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.909860 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-util\") pod \"b296384c-2609-4ce0-9a98-461cddd3b0cb\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " Sep 17 01:48:30.909944 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.909898 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-bundle\") pod \"b296384c-2609-4ce0-9a98-461cddd3b0cb\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " Sep 17 01:48:30.909944 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.909918 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-util\") pod \"817ba496-7206-4225-aefc-f90ba58284d3\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " Sep 17 01:48:30.909944 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.909946 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hnzj\" (UniqueName: \"kubernetes.io/projected/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-kube-api-access-7hnzj\") pod \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " Sep 17 01:48:30.910267 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.909987 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-bundle\") pod \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " Sep 17 01:48:30.910267 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.910006 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-util\") pod \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " Sep 17 01:48:30.910267 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.910051 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkwzz\" (UniqueName: \"kubernetes.io/projected/b296384c-2609-4ce0-9a98-461cddd3b0cb-kube-api-access-jkwzz\") pod \"b296384c-2609-4ce0-9a98-461cddd3b0cb\" (UID: \"b296384c-2609-4ce0-9a98-461cddd3b0cb\") " Sep 17 01:48:30.910417 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.910323 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w22p2\" (UniqueName: \"kubernetes.io/projected/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-kube-api-access-w22p2\") pod \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\" (UID: \"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de\") " Sep 17 01:48:30.910417 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.910365 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-bundle\") pod \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " Sep 17 01:48:30.910417 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.910389 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-util\") pod \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\" (UID: \"a1d07e4b-9239-4464-8bd3-6707c7ad51c5\") " Sep 17 01:48:30.910561 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.910431 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8b42\" (UniqueName: \"kubernetes.io/projected/817ba496-7206-4225-aefc-f90ba58284d3-kube-api-access-g8b42\") pod \"817ba496-7206-4225-aefc-f90ba58284d3\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " Sep 17 01:48:30.910561 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.910477 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-bundle\") pod \"817ba496-7206-4225-aefc-f90ba58284d3\" (UID: \"817ba496-7206-4225-aefc-f90ba58284d3\") " Sep 17 01:48:30.910766 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.910737 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-bundle" (OuterVolumeSpecName: "bundle") pod "b296384c-2609-4ce0-9a98-461cddd3b0cb" (UID: "b296384c-2609-4ce0-9a98-461cddd3b0cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:48:30.910865 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.910843 2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-bundle\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:30.911216 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.911197 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-bundle" (OuterVolumeSpecName: "bundle") pod "a1d07e4b-9239-4464-8bd3-6707c7ad51c5" (UID: "a1d07e4b-9239-4464-8bd3-6707c7ad51c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:48:30.911360 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.911282 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-bundle" (OuterVolumeSpecName: "bundle") pod "d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" (UID: "d8000fe1-9b79-4d1e-a74a-4fc9fdb960de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:48:30.912238 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.912028 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-bundle" (OuterVolumeSpecName: "bundle") pod "817ba496-7206-4225-aefc-f90ba58284d3" (UID: "817ba496-7206-4225-aefc-f90ba58284d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:48:30.913292 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.913264 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-kube-api-access-7hnzj" (OuterVolumeSpecName: "kube-api-access-7hnzj") pod "a1d07e4b-9239-4464-8bd3-6707c7ad51c5" (UID: "a1d07e4b-9239-4464-8bd3-6707c7ad51c5"). InnerVolumeSpecName "kube-api-access-7hnzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:48:30.913751 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.913560 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b296384c-2609-4ce0-9a98-461cddd3b0cb-kube-api-access-jkwzz" (OuterVolumeSpecName: "kube-api-access-jkwzz") pod "b296384c-2609-4ce0-9a98-461cddd3b0cb" (UID: "b296384c-2609-4ce0-9a98-461cddd3b0cb"). InnerVolumeSpecName "kube-api-access-jkwzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:48:30.913839 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.913766 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817ba496-7206-4225-aefc-f90ba58284d3-kube-api-access-g8b42" (OuterVolumeSpecName: "kube-api-access-g8b42") pod "817ba496-7206-4225-aefc-f90ba58284d3" (UID: "817ba496-7206-4225-aefc-f90ba58284d3"). InnerVolumeSpecName "kube-api-access-g8b42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:48:30.914077 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.914054 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-kube-api-access-w22p2" (OuterVolumeSpecName: "kube-api-access-w22p2") pod "d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" (UID: "d8000fe1-9b79-4d1e-a74a-4fc9fdb960de"). InnerVolumeSpecName "kube-api-access-w22p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:48:30.919134 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.919104 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-util" (OuterVolumeSpecName: "util") pod "817ba496-7206-4225-aefc-f90ba58284d3" (UID: "817ba496-7206-4225-aefc-f90ba58284d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:48:30.919627 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.919602 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-util" (OuterVolumeSpecName: "util") pod "b296384c-2609-4ce0-9a98-461cddd3b0cb" (UID: "b296384c-2609-4ce0-9a98-461cddd3b0cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:48:30.919781 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.919756 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-util" (OuterVolumeSpecName: "util") pod "a1d07e4b-9239-4464-8bd3-6707c7ad51c5" (UID: "a1d07e4b-9239-4464-8bd3-6707c7ad51c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:48:30.919937 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:30.919914 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-util" (OuterVolumeSpecName: "util") pod "d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" (UID: "d8000fe1-9b79-4d1e-a74a-4fc9fdb960de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:48:31.011870 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011831 2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-util\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.011870 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011864 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hnzj\" (UniqueName: \"kubernetes.io/projected/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-kube-api-access-7hnzj\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.011870 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011878 2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-bundle\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.012117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011889 2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-util\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.012117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011900 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkwzz\" (UniqueName: \"kubernetes.io/projected/b296384c-2609-4ce0-9a98-461cddd3b0cb-kube-api-access-jkwzz\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.012117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011911 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w22p2\" (UniqueName: \"kubernetes.io/projected/d8000fe1-9b79-4d1e-a74a-4fc9fdb960de-kube-api-access-w22p2\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.012117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011922 2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-bundle\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.012117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011932 2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1d07e4b-9239-4464-8bd3-6707c7ad51c5-util\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.012117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011942 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8b42\" (UniqueName: \"kubernetes.io/projected/817ba496-7206-4225-aefc-f90ba58284d3-kube-api-access-g8b42\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.012117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011953 2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/817ba496-7206-4225-aefc-f90ba58284d3-bundle\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.012117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.011963 2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b296384c-2609-4ce0-9a98-461cddd3b0cb-util\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:48:31.639131 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.639098 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" Sep 17 01:48:31.639313 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.639097 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68rpvx" event={"ID":"a1d07e4b-9239-4464-8bd3-6707c7ad51c5","Type":"ContainerDied","Data":"f9d652b9db25e5652bf364e4584f602ac33b3eb96f6ff0292561f1a7aac15658"} Sep 17 01:48:31.639313 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.639214 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9d652b9db25e5652bf364e4584f602ac33b3eb96f6ff0292561f1a7aac15658" Sep 17 01:48:31.641105 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.641088 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" Sep 17 01:48:31.641105 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.641103 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gc5hj" event={"ID":"d8000fe1-9b79-4d1e-a74a-4fc9fdb960de","Type":"ContainerDied","Data":"1b352b3586ad316e7e64948b688e8d0209da3eb2403556383e27c2b81b65c2ed"} Sep 17 01:48:31.641249 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.641133 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b352b3586ad316e7e64948b688e8d0209da3eb2403556383e27c2b81b65c2ed" Sep 17 01:48:31.643745 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.643723 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" Sep 17 01:48:31.643868 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.643726 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm5zs" event={"ID":"b296384c-2609-4ce0-9a98-461cddd3b0cb","Type":"ContainerDied","Data":"1d6c3b3ef7263988e19adc87190e2674911b2b5cd02b4287742d8287f7dd9f3c"} Sep 17 01:48:31.643868 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.643855 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d6c3b3ef7263988e19adc87190e2674911b2b5cd02b4287742d8287f7dd9f3c" Sep 17 01:48:31.645827 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.645806 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" event={"ID":"817ba496-7206-4225-aefc-f90ba58284d3","Type":"ContainerDied","Data":"55acd4150730ad7aaf7e40f03031360695e51f6b66b13162cfa952526268e203"} Sep 17 01:48:31.645919 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.645840 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55acd4150730ad7aaf7e40f03031360695e51f6b66b13162cfa952526268e203" Sep 17 01:48:31.645919 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:48:31.645846 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch5jdl" Sep 17 01:49:07.050121 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050079 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm"] Sep 17 01:49:07.050760 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050691 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817ba496-7206-4225-aefc-f90ba58284d3" containerName="util" Sep 17 01:49:07.050760 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050712 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="817ba496-7206-4225-aefc-f90ba58284d3" containerName="util" Sep 17 01:49:07.050760 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050724 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" containerName="extract" Sep 17 01:49:07.050760 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050733 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" containerName="extract" Sep 17 01:49:07.050760 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050745 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817ba496-7206-4225-aefc-f90ba58284d3" containerName="extract" Sep 17 01:49:07.050760 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050753 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="817ba496-7206-4225-aefc-f90ba58284d3" containerName="extract" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050772 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d07e4b-9239-4464-8bd3-6707c7ad51c5" containerName="util" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050780 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d07e4b-9239-4464-8bd3-6707c7ad51c5" containerName="util" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050797 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d07e4b-9239-4464-8bd3-6707c7ad51c5" containerName="pull" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050806 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d07e4b-9239-4464-8bd3-6707c7ad51c5" containerName="pull" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050815 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d07e4b-9239-4464-8bd3-6707c7ad51c5" containerName="extract" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050823 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d07e4b-9239-4464-8bd3-6707c7ad51c5" containerName="extract" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050835 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b296384c-2609-4ce0-9a98-461cddd3b0cb" containerName="pull" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050842 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b296384c-2609-4ce0-9a98-461cddd3b0cb" containerName="pull" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050858 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" containerName="util" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050866 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" containerName="util" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050877 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817ba496-7206-4225-aefc-f90ba58284d3" containerName="pull" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050885 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="817ba496-7206-4225-aefc-f90ba58284d3" containerName="pull" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050896 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b296384c-2609-4ce0-9a98-461cddd3b0cb" containerName="util" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050904 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b296384c-2609-4ce0-9a98-461cddd3b0cb" containerName="util" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050913 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" containerName="pull" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050922 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" containerName="pull" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050949 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b296384c-2609-4ce0-9a98-461cddd3b0cb" containerName="extract" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.050958 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b296384c-2609-4ce0-9a98-461cddd3b0cb" containerName="extract" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.051064 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d07e4b-9239-4464-8bd3-6707c7ad51c5" containerName="extract" Sep 17 01:49:07.051074 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.051076 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="817ba496-7206-4225-aefc-f90ba58284d3" containerName="extract" Sep 17 01:49:07.052004 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.051086 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b296384c-2609-4ce0-9a98-461cddd3b0cb" containerName="extract" Sep 17 01:49:07.052004 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.051098 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8000fe1-9b79-4d1e-a74a-4fc9fdb960de" containerName="extract" Sep 17 01:49:07.055976 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.055946 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.058928 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.058750 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 17 01:49:07.058928 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.058891 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 17 01:49:07.059719 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.059692 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 17 01:49:07.059847 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.059693 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 17 01:49:07.059906 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.059696 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6khtx\"" Sep 17 01:49:07.062560 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.062536 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm"] Sep 17 01:49:07.146528 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.146478 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pzjmm\" (UID: \"1d8b47d2-d6ac-4e7f-a157-df5e9112a150\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.146528 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.146532 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv6f7\" (UniqueName: \"kubernetes.io/projected/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-kube-api-access-wv6f7\") pod \"kuadrant-console-plugin-fbdd4f445-pzjmm\" (UID: \"1d8b47d2-d6ac-4e7f-a157-df5e9112a150\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.146830 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.146621 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pzjmm\" (UID: \"1d8b47d2-d6ac-4e7f-a157-df5e9112a150\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.247337 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.247304 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pzjmm\" (UID: \"1d8b47d2-d6ac-4e7f-a157-df5e9112a150\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.247546 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.247412 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pzjmm\" (UID: \"1d8b47d2-d6ac-4e7f-a157-df5e9112a150\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.247546 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.247438 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv6f7\" (UniqueName: \"kubernetes.io/projected/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-kube-api-access-wv6f7\") pod \"kuadrant-console-plugin-fbdd4f445-pzjmm\" (UID: \"1d8b47d2-d6ac-4e7f-a157-df5e9112a150\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.247737 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:49:07.247549 2586 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 17 01:49:07.247737 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:49:07.247650 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-plugin-serving-cert podName:1d8b47d2-d6ac-4e7f-a157-df5e9112a150 nodeName:}" failed. No retries permitted until 2026-09-17 01:49:07.747631056 +0000 UTC m=+711.504398292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-pzjmm" (UID: "1d8b47d2-d6ac-4e7f-a157-df5e9112a150") : secret "plugin-serving-cert" not found Sep 17 01:49:07.248082 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.248056 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pzjmm\" (UID: \"1d8b47d2-d6ac-4e7f-a157-df5e9112a150\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.258004 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.257978 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv6f7\" (UniqueName: \"kubernetes.io/projected/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-kube-api-access-wv6f7\") pod \"kuadrant-console-plugin-fbdd4f445-pzjmm\" (UID: \"1d8b47d2-d6ac-4e7f-a157-df5e9112a150\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.752539 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.752442 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pzjmm\" (UID: \"1d8b47d2-d6ac-4e7f-a157-df5e9112a150\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.755000 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.754972 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d8b47d2-d6ac-4e7f-a157-df5e9112a150-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pzjmm\" (UID: \"1d8b47d2-d6ac-4e7f-a157-df5e9112a150\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:07.966485 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:07.966453 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" Sep 17 01:49:08.239684 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:08.239656 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm"] Sep 17 01:49:08.241660 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:49:08.241635 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d8b47d2_d6ac_4e7f_a157_df5e9112a150.slice/crio-fce060c049383f4da54062b848c40de8cb774ebd35425908f75d2f4c6b5eddce WatchSource:0}: Error finding container fce060c049383f4da54062b848c40de8cb774ebd35425908f75d2f4c6b5eddce: Status 404 returned error can't find the container with id fce060c049383f4da54062b848c40de8cb774ebd35425908f75d2f4c6b5eddce Sep 17 01:49:08.243123 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:08.243104 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:49:08.854969 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:08.854936 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" event={"ID":"1d8b47d2-d6ac-4e7f-a157-df5e9112a150","Type":"ContainerStarted","Data":"fce060c049383f4da54062b848c40de8cb774ebd35425908f75d2f4c6b5eddce"} Sep 17 01:49:12.882139 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:12.882106 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" event={"ID":"1d8b47d2-d6ac-4e7f-a157-df5e9112a150","Type":"ContainerStarted","Data":"45dae98cdd81fbf2718d281731614d6e630f76b666529e628e56ac5600d10df9"} Sep 17 01:49:12.899503 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:12.899450 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pzjmm" podStartSLOduration=1.344853978 podStartE2EDuration="5.899433564s" totalImagesPullingTime="4.554579586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:49:12.897173716 +0000 UTC m=+716.653940961" watchObservedRunningTime="2026-09-17 01:49:12.899433564 +0000 UTC m=+716.656200851" Sep 17 01:49:47.910876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:47.910839 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj"] Sep 17 01:49:47.933295 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:47.933261 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj"] Sep 17 01:49:47.933447 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:47.933388 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:47.935940 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:47.935919 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pwbx8\"" Sep 17 01:49:48.031515 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.031461 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.031515 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.031519 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.031753 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.031575 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.031753 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.031616 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.031753 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.031686 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.031753 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.031723 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlg62\" (UniqueName: \"kubernetes.io/projected/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-kube-api-access-hlg62\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.031887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.031754 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.031887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.031828 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.031887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.031849 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.132956 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.132909 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.132956 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.132961 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlg62\" (UniqueName: \"kubernetes.io/projected/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-kube-api-access-hlg62\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133219 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.132987 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133219 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133039 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133219 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133054 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133219 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133073 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133219 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133088 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133479 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133223 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133479 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133271 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133479 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133441 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133626 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133500 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133626 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133555 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.133706 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133654 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.134002 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.133984 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.135525 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.135497 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.135751 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.135734 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.140889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.140865 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.141044 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.141028 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlg62\" (UniqueName: \"kubernetes.io/projected/4a36f446-3677-49e8-b0a6-e0c1eccfe23d-kube-api-access-hlg62\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d8gtj\" (UID: \"4a36f446-3677-49e8-b0a6-e0c1eccfe23d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.244601 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.244468 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:48.520804 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.520773 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj"] Sep 17 01:49:48.521999 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:49:48.521975 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a36f446_3677_49e8_b0a6_e0c1eccfe23d.slice/crio-5be69e9dd6d0fb824096b5c35ddc283d3ee557a4fb1dc083197dacfc5c638f1a WatchSource:0}: Error finding container 5be69e9dd6d0fb824096b5c35ddc283d3ee557a4fb1dc083197dacfc5c638f1a: Status 404 returned error can't find the container with id 5be69e9dd6d0fb824096b5c35ddc283d3ee557a4fb1dc083197dacfc5c638f1a Sep 17 01:49:48.524280 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.524242 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:49:48.524360 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.524305 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:49:48.524360 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:48.524338 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:49:49.086007 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:49.085965 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" event={"ID":"4a36f446-3677-49e8-b0a6-e0c1eccfe23d","Type":"ContainerStarted","Data":"f685791854e45838c99edb52e5d4c2b73ee35d3f35c70da2b4eb252a392774ee"} Sep 17 01:49:49.086389 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:49.086029 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" event={"ID":"4a36f446-3677-49e8-b0a6-e0c1eccfe23d","Type":"ContainerStarted","Data":"5be69e9dd6d0fb824096b5c35ddc283d3ee557a4fb1dc083197dacfc5c638f1a"} Sep 17 01:49:49.105481 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:49.105430 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" podStartSLOduration=2.105413546 podStartE2EDuration="2.105413546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:49:49.104452409 +0000 UTC m=+752.861219666" watchObservedRunningTime="2026-09-17 01:49:49.105413546 +0000 UTC m=+752.862180788" Sep 17 01:49:49.245552 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:49.245516 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:49.247131 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:49.247105 2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d8gtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 17 01:49:49.247244 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:49.247159 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" podUID="4a36f446-3677-49e8-b0a6-e0c1eccfe23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 17 01:49:50.248958 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:50.248930 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:51.097544 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:51.097512 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:49:51.098624 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:49:51.098601 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:50:00.473266 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.473227 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h24xn"] Sep 17 01:50:00.477057 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.477037 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" Sep 17 01:50:00.479420 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.479382 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fc58z\"" Sep 17 01:50:00.483615 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.483565 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h24xn"] Sep 17 01:50:00.549957 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.549918 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn8cs\" (UniqueName: \"kubernetes.io/projected/dc96c1f3-b399-407e-9bfd-2ffd5595a436-kube-api-access-cn8cs\") pod \"authorino-7bcfd8f9cd-h24xn\" (UID: \"dc96c1f3-b399-407e-9bfd-2ffd5595a436\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" Sep 17 01:50:00.596317 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.596278 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-jfbjh"] Sep 17 01:50:00.599803 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.599784 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jfbjh" Sep 17 01:50:00.607161 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.607127 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jfbjh"] Sep 17 01:50:00.650902 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.650862 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn8cs\" (UniqueName: \"kubernetes.io/projected/dc96c1f3-b399-407e-9bfd-2ffd5595a436-kube-api-access-cn8cs\") pod \"authorino-7bcfd8f9cd-h24xn\" (UID: \"dc96c1f3-b399-407e-9bfd-2ffd5595a436\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" Sep 17 01:50:00.661267 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.661236 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn8cs\" (UniqueName: \"kubernetes.io/projected/dc96c1f3-b399-407e-9bfd-2ffd5595a436-kube-api-access-cn8cs\") pod \"authorino-7bcfd8f9cd-h24xn\" (UID: \"dc96c1f3-b399-407e-9bfd-2ffd5595a436\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" Sep 17 01:50:00.752256 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.752150 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgsl\" (UniqueName: \"kubernetes.io/projected/81155b88-9d23-468a-8fd9-c32359534a6c-kube-api-access-8wgsl\") pod \"authorino-59c974bd48-jfbjh\" (UID: \"81155b88-9d23-468a-8fd9-c32359534a6c\") " pod="kuadrant-system/authorino-59c974bd48-jfbjh" Sep 17 01:50:00.787345 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.787308 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" Sep 17 01:50:00.853838 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.853794 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgsl\" (UniqueName: \"kubernetes.io/projected/81155b88-9d23-468a-8fd9-c32359534a6c-kube-api-access-8wgsl\") pod \"authorino-59c974bd48-jfbjh\" (UID: \"81155b88-9d23-468a-8fd9-c32359534a6c\") " pod="kuadrant-system/authorino-59c974bd48-jfbjh" Sep 17 01:50:00.863854 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.863828 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgsl\" (UniqueName: \"kubernetes.io/projected/81155b88-9d23-468a-8fd9-c32359534a6c-kube-api-access-8wgsl\") pod \"authorino-59c974bd48-jfbjh\" (UID: \"81155b88-9d23-468a-8fd9-c32359534a6c\") " pod="kuadrant-system/authorino-59c974bd48-jfbjh" Sep 17 01:50:00.910444 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:00.910403 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jfbjh" Sep 17 01:50:01.059750 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:01.059723 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h24xn"] Sep 17 01:50:01.062379 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:50:01.062352 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc96c1f3_b399_407e_9bfd_2ffd5595a436.slice/crio-adee8617a7a045e8862dbe46cfb3360a222ad50ce91fe3fe3147207729684293 WatchSource:0}: Error finding container adee8617a7a045e8862dbe46cfb3360a222ad50ce91fe3fe3147207729684293: Status 404 returned error can't find the container with id adee8617a7a045e8862dbe46cfb3360a222ad50ce91fe3fe3147207729684293 Sep 17 01:50:01.151198 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:01.151160 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" event={"ID":"dc96c1f3-b399-407e-9bfd-2ffd5595a436","Type":"ContainerStarted","Data":"adee8617a7a045e8862dbe46cfb3360a222ad50ce91fe3fe3147207729684293"} Sep 17 01:50:01.184261 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:01.184225 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jfbjh"] Sep 17 01:50:01.185916 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:50:01.185890 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81155b88_9d23_468a_8fd9_c32359534a6c.slice/crio-2224fd0f1789ee22595ff60335c5f56bf0920c7800505cafbdf08d2c26436ed5 WatchSource:0}: Error finding container 2224fd0f1789ee22595ff60335c5f56bf0920c7800505cafbdf08d2c26436ed5: Status 404 returned error can't find the container with id 2224fd0f1789ee22595ff60335c5f56bf0920c7800505cafbdf08d2c26436ed5 Sep 17 01:50:02.164330 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:02.164277 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jfbjh" event={"ID":"81155b88-9d23-468a-8fd9-c32359534a6c","Type":"ContainerStarted","Data":"2224fd0f1789ee22595ff60335c5f56bf0920c7800505cafbdf08d2c26436ed5"} Sep 17 01:50:04.178396 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:04.178352 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" event={"ID":"dc96c1f3-b399-407e-9bfd-2ffd5595a436","Type":"ContainerStarted","Data":"72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc"} Sep 17 01:50:04.180068 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:04.180042 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jfbjh" event={"ID":"81155b88-9d23-468a-8fd9-c32359534a6c","Type":"ContainerStarted","Data":"eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0"} Sep 17 01:50:04.200253 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:04.200198 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" podStartSLOduration=1.291363023 podStartE2EDuration="4.200182374s" totalImagesPullingTime="2.908819351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:04.198359931 +0000 UTC m=+767.955127174" watchObservedRunningTime="2026-09-17 01:50:04.200182374 +0000 UTC m=+767.956949618" Sep 17 01:50:04.229922 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:04.229812 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-jfbjh" podStartSLOduration=1.4330621749999999 podStartE2EDuration="4.229795477s" totalImagesPullingTime="2.796733302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:04.22846092 +0000 UTC m=+767.985228165" watchObservedRunningTime="2026-09-17 01:50:04.229795477 +0000 UTC m=+767.986562719" Sep 17 01:50:04.275989 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:04.275950 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h24xn"] Sep 17 01:50:06.190865 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:06.190816 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" podUID="dc96c1f3-b399-407e-9bfd-2ffd5595a436" containerName="authorino" containerID="cri-o://72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc" gracePeriod=30 Sep 17 01:50:06.441492 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:06.441437 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" Sep 17 01:50:06.608307 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:06.608270 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn8cs\" (UniqueName: \"kubernetes.io/projected/dc96c1f3-b399-407e-9bfd-2ffd5595a436-kube-api-access-cn8cs\") pod \"dc96c1f3-b399-407e-9bfd-2ffd5595a436\" (UID: \"dc96c1f3-b399-407e-9bfd-2ffd5595a436\") " Sep 17 01:50:06.610672 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:06.610645 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc96c1f3-b399-407e-9bfd-2ffd5595a436-kube-api-access-cn8cs" (OuterVolumeSpecName: "kube-api-access-cn8cs") pod "dc96c1f3-b399-407e-9bfd-2ffd5595a436" (UID: "dc96c1f3-b399-407e-9bfd-2ffd5595a436"). InnerVolumeSpecName "kube-api-access-cn8cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:50:06.709624 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:06.709518 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn8cs\" (UniqueName: \"kubernetes.io/projected/dc96c1f3-b399-407e-9bfd-2ffd5595a436-kube-api-access-cn8cs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:50:07.197840 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:07.197800 2586 generic.go:356] "Generic (PLEG): container finished" podID="dc96c1f3-b399-407e-9bfd-2ffd5595a436" containerID="72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc" exitCode=0 Sep 17 01:50:07.198277 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:07.197851 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" Sep 17 01:50:07.198277 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:07.197865 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" event={"ID":"dc96c1f3-b399-407e-9bfd-2ffd5595a436","Type":"ContainerDied","Data":"72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc"} Sep 17 01:50:07.198277 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:07.197896 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h24xn" event={"ID":"dc96c1f3-b399-407e-9bfd-2ffd5595a436","Type":"ContainerDied","Data":"adee8617a7a045e8862dbe46cfb3360a222ad50ce91fe3fe3147207729684293"} Sep 17 01:50:07.198277 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:07.197912 2586 scope.go:117] "RemoveContainer" containerID="72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc" Sep 17 01:50:07.206716 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:07.206693 2586 scope.go:117] "RemoveContainer" containerID="72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc" Sep 17 01:50:07.206969 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:50:07.206948 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc\": container with ID starting with 72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc not found: ID does not exist" containerID="72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc" Sep 17 01:50:07.207036 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:07.206979 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc"} err="failed to get container status \"72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc\": rpc error: code = NotFound desc = could not find container \"72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc\": container with ID starting with 72baa227164a58d80fb175a7dd1759650b070838392385b0d1fc646eb77816dc not found: ID does not exist" Sep 17 01:50:07.214285 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:07.214263 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h24xn"] Sep 17 01:50:07.217637 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:07.217617 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h24xn"] Sep 17 01:50:08.767499 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:08.767468 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc96c1f3-b399-407e-9bfd-2ffd5595a436" path="/var/lib/kubelet/pods/dc96c1f3-b399-407e-9bfd-2ffd5595a436/volumes" Sep 17 01:50:29.869303 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.869269 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-x7hbw"] Sep 17 01:50:29.869798 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.869667 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc96c1f3-b399-407e-9bfd-2ffd5595a436" containerName="authorino" Sep 17 01:50:29.869798 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.869680 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc96c1f3-b399-407e-9bfd-2ffd5595a436" containerName="authorino" Sep 17 01:50:29.869798 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.869756 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc96c1f3-b399-407e-9bfd-2ffd5595a436" containerName="authorino" Sep 17 01:50:29.877764 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.877734 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x7hbw" Sep 17 01:50:29.878476 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.878454 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-x7hbw"] Sep 17 01:50:29.916331 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.916299 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-x7hbw"] Sep 17 01:50:29.916633 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:50:29.916614 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-p4cfp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-x7hbw" podUID="a55ee858-1396-4f86-9a97-0b3a4423c0c8" Sep 17 01:50:29.921100 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.921076 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cfp\" (UniqueName: \"kubernetes.io/projected/a55ee858-1396-4f86-9a97-0b3a4423c0c8-kube-api-access-p4cfp\") pod \"authorino-557f876f69-x7hbw\" (UID: \"a55ee858-1396-4f86-9a97-0b3a4423c0c8\") " pod="kuadrant-system/authorino-557f876f69-x7hbw" Sep 17 01:50:29.945471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.945442 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p5j2d"] Sep 17 01:50:29.949572 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.949556 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" Sep 17 01:50:29.951862 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.951843 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 17 01:50:29.955189 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:29.955168 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p5j2d"] Sep 17 01:50:30.021514 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.021477 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/07f9f876-4332-45ca-b4d6-4fe360923338-tls-cert\") pod \"authorino-6d4dfc649f-p5j2d\" (UID: \"07f9f876-4332-45ca-b4d6-4fe360923338\") " pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" Sep 17 01:50:30.021721 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.021619 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cfp\" (UniqueName: \"kubernetes.io/projected/a55ee858-1396-4f86-9a97-0b3a4423c0c8-kube-api-access-p4cfp\") pod \"authorino-557f876f69-x7hbw\" (UID: \"a55ee858-1396-4f86-9a97-0b3a4423c0c8\") " pod="kuadrant-system/authorino-557f876f69-x7hbw" Sep 17 01:50:30.021721 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.021655 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mwl2\" (UniqueName: \"kubernetes.io/projected/07f9f876-4332-45ca-b4d6-4fe360923338-kube-api-access-5mwl2\") pod \"authorino-6d4dfc649f-p5j2d\" (UID: \"07f9f876-4332-45ca-b4d6-4fe360923338\") " pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" Sep 17 01:50:30.030320 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.030294 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cfp\" (UniqueName: \"kubernetes.io/projected/a55ee858-1396-4f86-9a97-0b3a4423c0c8-kube-api-access-p4cfp\") pod \"authorino-557f876f69-x7hbw\" (UID: \"a55ee858-1396-4f86-9a97-0b3a4423c0c8\") " pod="kuadrant-system/authorino-557f876f69-x7hbw" Sep 17 01:50:30.122870 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.122773 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mwl2\" (UniqueName: \"kubernetes.io/projected/07f9f876-4332-45ca-b4d6-4fe360923338-kube-api-access-5mwl2\") pod \"authorino-6d4dfc649f-p5j2d\" (UID: \"07f9f876-4332-45ca-b4d6-4fe360923338\") " pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" Sep 17 01:50:30.122870 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.122818 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/07f9f876-4332-45ca-b4d6-4fe360923338-tls-cert\") pod \"authorino-6d4dfc649f-p5j2d\" (UID: \"07f9f876-4332-45ca-b4d6-4fe360923338\") " pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" Sep 17 01:50:30.124090 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.124062 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p5j2d"] Sep 17 01:50:30.124324 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:50:30.124305 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5mwl2 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" podUID="07f9f876-4332-45ca-b4d6-4fe360923338" Sep 17 01:50:30.125437 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.125409 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/07f9f876-4332-45ca-b4d6-4fe360923338-tls-cert\") pod \"authorino-6d4dfc649f-p5j2d\" (UID: \"07f9f876-4332-45ca-b4d6-4fe360923338\") " pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" Sep 17 01:50:30.137633 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.137613 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mwl2\" (UniqueName: \"kubernetes.io/projected/07f9f876-4332-45ca-b4d6-4fe360923338-kube-api-access-5mwl2\") pod \"authorino-6d4dfc649f-p5j2d\" (UID: \"07f9f876-4332-45ca-b4d6-4fe360923338\") " pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" Sep 17 01:50:30.150358 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.150331 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7dc9999c6b-2x2w7"] Sep 17 01:50:30.154591 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.154564 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" Sep 17 01:50:30.160680 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.160660 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dc9999c6b-2x2w7"] Sep 17 01:50:30.223927 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.223892 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-tls-cert\") pod \"authorino-7dc9999c6b-2x2w7\" (UID: \"f8c05c0f-c60e-458c-baba-6ee6355ab0c3\") " pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" Sep 17 01:50:30.224128 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.224013 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87tp\" (UniqueName: \"kubernetes.io/projected/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-kube-api-access-d87tp\") pod \"authorino-7dc9999c6b-2x2w7\" (UID: \"f8c05c0f-c60e-458c-baba-6ee6355ab0c3\") " pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" Sep 17 01:50:30.324370 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.324338 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" Sep 17 01:50:30.324559 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.324345 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x7hbw" Sep 17 01:50:30.324659 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.324631 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-tls-cert\") pod \"authorino-7dc9999c6b-2x2w7\" (UID: \"f8c05c0f-c60e-458c-baba-6ee6355ab0c3\") " pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" Sep 17 01:50:30.324757 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.324741 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d87tp\" (UniqueName: \"kubernetes.io/projected/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-kube-api-access-d87tp\") pod \"authorino-7dc9999c6b-2x2w7\" (UID: \"f8c05c0f-c60e-458c-baba-6ee6355ab0c3\") " pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" Sep 17 01:50:30.327137 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.327116 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-tls-cert\") pod \"authorino-7dc9999c6b-2x2w7\" (UID: \"f8c05c0f-c60e-458c-baba-6ee6355ab0c3\") " pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" Sep 17 01:50:30.332117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.332092 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87tp\" (UniqueName: \"kubernetes.io/projected/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-kube-api-access-d87tp\") pod \"authorino-7dc9999c6b-2x2w7\" (UID: \"f8c05c0f-c60e-458c-baba-6ee6355ab0c3\") " pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" Sep 17 01:50:30.350028 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.349985 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" Sep 17 01:50:30.353339 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.353320 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x7hbw" Sep 17 01:50:30.425471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.425391 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4cfp\" (UniqueName: \"kubernetes.io/projected/a55ee858-1396-4f86-9a97-0b3a4423c0c8-kube-api-access-p4cfp\") pod \"a55ee858-1396-4f86-9a97-0b3a4423c0c8\" (UID: \"a55ee858-1396-4f86-9a97-0b3a4423c0c8\") " Sep 17 01:50:30.425471 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.425448 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/07f9f876-4332-45ca-b4d6-4fe360923338-tls-cert\") pod \"07f9f876-4332-45ca-b4d6-4fe360923338\" (UID: \"07f9f876-4332-45ca-b4d6-4fe360923338\") " Sep 17 01:50:30.425714 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.425473 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mwl2\" (UniqueName: \"kubernetes.io/projected/07f9f876-4332-45ca-b4d6-4fe360923338-kube-api-access-5mwl2\") pod \"07f9f876-4332-45ca-b4d6-4fe360923338\" (UID: \"07f9f876-4332-45ca-b4d6-4fe360923338\") " Sep 17 01:50:30.427627 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.427601 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f9f876-4332-45ca-b4d6-4fe360923338-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "07f9f876-4332-45ca-b4d6-4fe360923338" (UID: "07f9f876-4332-45ca-b4d6-4fe360923338"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:50:30.427715 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.427624 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55ee858-1396-4f86-9a97-0b3a4423c0c8-kube-api-access-p4cfp" (OuterVolumeSpecName: "kube-api-access-p4cfp") pod "a55ee858-1396-4f86-9a97-0b3a4423c0c8" (UID: "a55ee858-1396-4f86-9a97-0b3a4423c0c8"). InnerVolumeSpecName "kube-api-access-p4cfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:50:30.427715 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.427678 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f9f876-4332-45ca-b4d6-4fe360923338-kube-api-access-5mwl2" (OuterVolumeSpecName: "kube-api-access-5mwl2") pod "07f9f876-4332-45ca-b4d6-4fe360923338" (UID: "07f9f876-4332-45ca-b4d6-4fe360923338"). InnerVolumeSpecName "kube-api-access-5mwl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:50:30.465014 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.464987 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" Sep 17 01:50:30.526989 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.526918 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4cfp\" (UniqueName: \"kubernetes.io/projected/a55ee858-1396-4f86-9a97-0b3a4423c0c8-kube-api-access-p4cfp\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:50:30.526989 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.526948 2586 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/07f9f876-4332-45ca-b4d6-4fe360923338-tls-cert\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:50:30.526989 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.526963 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mwl2\" (UniqueName: \"kubernetes.io/projected/07f9f876-4332-45ca-b4d6-4fe360923338-kube-api-access-5mwl2\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:50:30.734895 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:30.734867 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dc9999c6b-2x2w7"] Sep 17 01:50:30.736487 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:50:30.736461 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8c05c0f_c60e_458c_baba_6ee6355ab0c3.slice/crio-73f8ed5adb87a752e4bf1e9a4ee57d06c9c62fc2c46e27841c97eaca818b51f1 WatchSource:0}: Error finding container 73f8ed5adb87a752e4bf1e9a4ee57d06c9c62fc2c46e27841c97eaca818b51f1: Status 404 returned error can't find the container with id 73f8ed5adb87a752e4bf1e9a4ee57d06c9c62fc2c46e27841c97eaca818b51f1 Sep 17 01:50:31.333035 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.332943 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" event={"ID":"f8c05c0f-c60e-458c-baba-6ee6355ab0c3","Type":"ContainerStarted","Data":"1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff"} Sep 17 01:50:31.333035 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.333003 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" event={"ID":"f8c05c0f-c60e-458c-baba-6ee6355ab0c3","Type":"ContainerStarted","Data":"73f8ed5adb87a752e4bf1e9a4ee57d06c9c62fc2c46e27841c97eaca818b51f1"} Sep 17 01:50:31.333035 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.333023 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p5j2d" Sep 17 01:50:31.333740 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.333292 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x7hbw" Sep 17 01:50:31.351297 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.351242 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" podStartSLOduration=0.869682441 podStartE2EDuration="1.35121364s" totalImagesPullingTime="481.531199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:31.347071472 +0000 UTC m=+795.103838721" watchObservedRunningTime="2026-09-17 01:50:31.35121364 +0000 UTC m=+795.107980888" Sep 17 01:50:31.379561 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.379528 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-x7hbw"] Sep 17 01:50:31.382826 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.382795 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jfbjh"] Sep 17 01:50:31.383015 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.382991 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-jfbjh" podUID="81155b88-9d23-468a-8fd9-c32359534a6c" containerName="authorino" containerID="cri-o://eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0" gracePeriod=30 Sep 17 01:50:31.386793 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.386718 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-x7hbw"] Sep 17 01:50:31.404334 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.404308 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p5j2d"] Sep 17 01:50:31.408658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.408622 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p5j2d"] Sep 17 01:50:31.626244 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.626217 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jfbjh" Sep 17 01:50:31.738700 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.738666 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wgsl\" (UniqueName: \"kubernetes.io/projected/81155b88-9d23-468a-8fd9-c32359534a6c-kube-api-access-8wgsl\") pod \"81155b88-9d23-468a-8fd9-c32359534a6c\" (UID: \"81155b88-9d23-468a-8fd9-c32359534a6c\") " Sep 17 01:50:31.740942 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.740920 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81155b88-9d23-468a-8fd9-c32359534a6c-kube-api-access-8wgsl" (OuterVolumeSpecName: "kube-api-access-8wgsl") pod "81155b88-9d23-468a-8fd9-c32359534a6c" (UID: "81155b88-9d23-468a-8fd9-c32359534a6c"). InnerVolumeSpecName "kube-api-access-8wgsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:50:31.840302 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:31.840263 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgsl\" (UniqueName: \"kubernetes.io/projected/81155b88-9d23-468a-8fd9-c32359534a6c-kube-api-access-8wgsl\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:50:32.342742 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.342619 2586 generic.go:356] "Generic (PLEG): container finished" podID="81155b88-9d23-468a-8fd9-c32359534a6c" containerID="eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0" exitCode=0 Sep 17 01:50:32.342742 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.342694 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jfbjh" Sep 17 01:50:32.342742 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.342715 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jfbjh" event={"ID":"81155b88-9d23-468a-8fd9-c32359534a6c","Type":"ContainerDied","Data":"eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0"} Sep 17 01:50:32.343503 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.342767 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jfbjh" event={"ID":"81155b88-9d23-468a-8fd9-c32359534a6c","Type":"ContainerDied","Data":"2224fd0f1789ee22595ff60335c5f56bf0920c7800505cafbdf08d2c26436ed5"} Sep 17 01:50:32.343503 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.342794 2586 scope.go:117] "RemoveContainer" containerID="eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0" Sep 17 01:50:32.353287 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.353256 2586 scope.go:117] "RemoveContainer" containerID="eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0" Sep 17 01:50:32.353683 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:50:32.353632 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0\": container with ID starting with eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0 not found: ID does not exist" containerID="eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0" Sep 17 01:50:32.353683 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.353668 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0"} err="failed to get container status \"eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0\": rpc error: code = NotFound desc = could not find container \"eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0\": container with ID starting with eda58127756124b33dc78141be6ea48cdfe044b72e5a0fa1741d08ca1f92cff0 not found: ID does not exist" Sep 17 01:50:32.367540 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.367511 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jfbjh"] Sep 17 01:50:32.370555 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.370533 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jfbjh"] Sep 17 01:50:32.768980 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.768950 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07f9f876-4332-45ca-b4d6-4fe360923338" path="/var/lib/kubelet/pods/07f9f876-4332-45ca-b4d6-4fe360923338/volumes" Sep 17 01:50:32.769177 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.769165 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81155b88-9d23-468a-8fd9-c32359534a6c" path="/var/lib/kubelet/pods/81155b88-9d23-468a-8fd9-c32359534a6c/volumes" Sep 17 01:50:32.769438 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:50:32.769427 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a55ee858-1396-4f86-9a97-0b3a4423c0c8" path="/var/lib/kubelet/pods/a55ee858-1396-4f86-9a97-0b3a4423c0c8/volumes" Sep 17 01:51:13.399773 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.399689 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz"] Sep 17 01:51:13.400245 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.400107 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81155b88-9d23-468a-8fd9-c32359534a6c" containerName="authorino" Sep 17 01:51:13.400245 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.400119 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="81155b88-9d23-468a-8fd9-c32359534a6c" containerName="authorino" Sep 17 01:51:13.400245 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.400192 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="81155b88-9d23-468a-8fd9-c32359534a6c" containerName="authorino" Sep 17 01:51:13.403415 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.403391 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:51:13.406101 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.406074 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 17 01:51:13.406299 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.406278 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8fgrd\"" Sep 17 01:51:13.412878 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.412848 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz"] Sep 17 01:51:13.425876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.424462 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjcd\" (UniqueName: \"kubernetes.io/projected/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-kube-api-access-qpjcd\") pod \"payload-pre-processing-59fbc47bf8-dxwzz\" (UID: \"e329c758-76f5-40a5-b5c1-86ff3a9d39c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:51:13.425876 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.424690 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dxwzz\" (UID: \"e329c758-76f5-40a5-b5c1-86ff3a9d39c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:51:13.525467 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.525419 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dxwzz\" (UID: \"e329c758-76f5-40a5-b5c1-86ff3a9d39c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:51:13.525819 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.525787 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjcd\" (UniqueName: \"kubernetes.io/projected/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-kube-api-access-qpjcd\") pod \"payload-pre-processing-59fbc47bf8-dxwzz\" (UID: \"e329c758-76f5-40a5-b5c1-86ff3a9d39c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:51:13.526190 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.526163 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dxwzz\" (UID: \"e329c758-76f5-40a5-b5c1-86ff3a9d39c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:51:13.539038 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.539007 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjcd\" (UniqueName: \"kubernetes.io/projected/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-kube-api-access-qpjcd\") pod \"payload-pre-processing-59fbc47bf8-dxwzz\" (UID: \"e329c758-76f5-40a5-b5c1-86ff3a9d39c2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:51:13.716431 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:13.716339 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:51:14.006020 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:14.005991 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz"] Sep 17 01:51:14.008521 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:51:14.008493 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode329c758_76f5_40a5_b5c1_86ff3a9d39c2.slice/crio-892cf0ea875d6dc59ede25d88c51c6b3dbf85255367b2a6277fda234f4c8bc0c WatchSource:0}: Error finding container 892cf0ea875d6dc59ede25d88c51c6b3dbf85255367b2a6277fda234f4c8bc0c: Status 404 returned error can't find the container with id 892cf0ea875d6dc59ede25d88c51c6b3dbf85255367b2a6277fda234f4c8bc0c Sep 17 01:51:14.589512 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:14.589478 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" event={"ID":"e329c758-76f5-40a5-b5c1-86ff3a9d39c2","Type":"ContainerStarted","Data":"892cf0ea875d6dc59ede25d88c51c6b3dbf85255367b2a6277fda234f4c8bc0c"} Sep 17 01:51:16.604351 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:16.604315 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" event={"ID":"e329c758-76f5-40a5-b5c1-86ff3a9d39c2","Type":"ContainerStarted","Data":"7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640"} Sep 17 01:51:16.604763 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:16.604374 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:51:16.623719 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:16.623653 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" podStartSLOduration=1.952625813 podStartE2EDuration="3.623630596s" totalImagesPullingTime="1.671004783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:51:16.619863572 +0000 UTC m=+840.376630843" watchObservedRunningTime="2026-09-17 01:51:16.623630596 +0000 UTC m=+840.380397843" Sep 17 01:51:22.139748 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:22.139702 2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d8gtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 01:51:22.140138 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:22.139764 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" podUID="4a36f446-3677-49e8-b0a6-e0c1eccfe23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 01:51:22.644031 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:22.643996 2586 generic.go:356] "Generic (PLEG): container finished" podID="4a36f446-3677-49e8-b0a6-e0c1eccfe23d" containerID="f685791854e45838c99edb52e5d4c2b73ee35d3f35c70da2b4eb252a392774ee" exitCode=0 Sep 17 01:51:22.644225 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:22.644049 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" event={"ID":"4a36f446-3677-49e8-b0a6-e0c1eccfe23d","Type":"ContainerDied","Data":"f685791854e45838c99edb52e5d4c2b73ee35d3f35c70da2b4eb252a392774ee"} Sep 17 01:51:22.644406 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:22.644389 2586 scope.go:117] "RemoveContainer" containerID="f685791854e45838c99edb52e5d4c2b73ee35d3f35c70da2b4eb252a392774ee" Sep 17 01:51:22.645373 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:22.645343 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:51:22.645437 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:22.645419 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:51:22.645485 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:22.645471 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:51:23.651521 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:23.651481 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" event={"ID":"4a36f446-3677-49e8-b0a6-e0c1eccfe23d","Type":"ContainerStarted","Data":"3b1f709a8b3881a42452e944da8a44774b7d8d9c9a6bf5f553cd34304c297697"} Sep 17 01:51:24.245330 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:24.245287 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:51:24.245547 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:24.245524 2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d8gtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 17 01:51:24.245615 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:24.245589 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" podUID="4a36f446-3677-49e8-b0a6-e0c1eccfe23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 17 01:51:25.245633 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:25.245566 2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d8gtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 17 01:51:25.246023 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:25.245669 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" podUID="4a36f446-3677-49e8-b0a6-e0c1eccfe23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 17 01:51:26.244942 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:26.244905 2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d8gtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 17 01:51:26.245253 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:26.245213 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" podUID="4a36f446-3677-49e8-b0a6-e0c1eccfe23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 17 01:51:27.245744 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:27.245705 2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d8gtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 17 01:51:27.246128 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:27.245777 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" podUID="4a36f446-3677-49e8-b0a6-e0c1eccfe23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 17 01:51:27.615200 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:27.615167 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:51:28.245310 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:28.245264 2586 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d8gtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 17 01:51:28.245533 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:28.245333 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" podUID="4a36f446-3677-49e8-b0a6-e0c1eccfe23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 17 01:51:29.249481 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:29.249451 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:51:29.249934 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:29.249787 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:51:29.250657 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:29.250639 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d8gtj" Sep 17 01:51:34.613226 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.613186 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx"] Sep 17 01:51:34.619385 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.619352 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.623020 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.622992 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 17 01:51:34.624092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.624068 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 17 01:51:34.624188 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.624096 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 17 01:51:34.624188 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.624076 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k5l9j\"" Sep 17 01:51:34.633157 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.633133 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx"] Sep 17 01:51:34.752220 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.752180 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.752220 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.752222 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.752426 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.752272 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6svb\" (UniqueName: \"kubernetes.io/projected/60040d37-ef8a-4b3a-8dd2-1035391f02ef-kube-api-access-k6svb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.752426 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.752300 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.752426 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.752317 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60040d37-ef8a-4b3a-8dd2-1035391f02ef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.752426 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.752351 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.853940 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.853900 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.853940 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.853936 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.854222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.854150 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6svb\" (UniqueName: \"kubernetes.io/projected/60040d37-ef8a-4b3a-8dd2-1035391f02ef-kube-api-access-k6svb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.854222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.854199 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.854366 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.854339 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.854426 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.854386 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.854510 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.854242 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60040d37-ef8a-4b3a-8dd2-1035391f02ef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.854608 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.854573 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.854948 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.854922 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.856493 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.856478 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60040d37-ef8a-4b3a-8dd2-1035391f02ef-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.856941 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.856920 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60040d37-ef8a-4b3a-8dd2-1035391f02ef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.861612 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.861557 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6svb\" (UniqueName: \"kubernetes.io/projected/60040d37-ef8a-4b3a-8dd2-1035391f02ef-kube-api-access-k6svb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx\" (UID: \"60040d37-ef8a-4b3a-8dd2-1035391f02ef\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:34.930686 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:34.930602 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:35.205097 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:35.205069 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx"] Sep 17 01:51:35.207383 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:51:35.207346 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60040d37_ef8a_4b3a_8dd2_1035391f02ef.slice/crio-06a317fb9a004799e513857a53519353b12151779d44e67a0dddbd9cf10a14f6 WatchSource:0}: Error finding container 06a317fb9a004799e513857a53519353b12151779d44e67a0dddbd9cf10a14f6: Status 404 returned error can't find the container with id 06a317fb9a004799e513857a53519353b12151779d44e67a0dddbd9cf10a14f6 Sep 17 01:51:35.725626 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:35.725563 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" event={"ID":"60040d37-ef8a-4b3a-8dd2-1035391f02ef","Type":"ContainerStarted","Data":"06a317fb9a004799e513857a53519353b12151779d44e67a0dddbd9cf10a14f6"} Sep 17 01:51:36.733015 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:36.732915 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" event={"ID":"60040d37-ef8a-4b3a-8dd2-1035391f02ef","Type":"ContainerStarted","Data":"08e3f43428baaf813bfefddbf0cd01d6a9efae98a396be92df16c332ecc3407d"} Sep 17 01:51:36.749386 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:36.749330 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" podStartSLOduration=1.483184342 podStartE2EDuration="2.7493125s" totalImagesPullingTime="1.266128158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:51:36.748227301 +0000 UTC m=+860.504994568" watchObservedRunningTime="2026-09-17 01:51:36.7493125 +0000 UTC m=+860.506079744" Sep 17 01:51:41.783706 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:41.783673 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr"] Sep 17 01:51:41.787235 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:41.787219 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:41.789720 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:41.789697 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 17 01:51:41.796994 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:41.796970 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr"] Sep 17 01:51:41.926374 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:41.926343 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:41.926534 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:41.926382 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:41.926534 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:41.926462 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:41.926534 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:41.926509 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr2pv\" (UniqueName: \"kubernetes.io/projected/ce109529-b5bb-4f22-a47c-ab2defa1b772-kube-api-access-mr2pv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:41.926673 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:41.926541 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:41.926673 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:41.926627 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce109529-b5bb-4f22-a47c-ab2defa1b772-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.027379 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.027342 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.027652 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.027408 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.027652 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.027431 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr2pv\" (UniqueName: \"kubernetes.io/projected/ce109529-b5bb-4f22-a47c-ab2defa1b772-kube-api-access-mr2pv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.027652 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.027456 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.027652 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.027488 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce109529-b5bb-4f22-a47c-ab2defa1b772-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.027652 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.027551 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.027903 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.027853 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.027949 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.027931 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.027991 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.027953 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.030214 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.030189 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce109529-b5bb-4f22-a47c-ab2defa1b772-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.030909 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.030890 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce109529-b5bb-4f22-a47c-ab2defa1b772-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.035528 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.035453 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr2pv\" (UniqueName: \"kubernetes.io/projected/ce109529-b5bb-4f22-a47c-ab2defa1b772-kube-api-access-mr2pv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bldfr\" (UID: \"ce109529-b5bb-4f22-a47c-ab2defa1b772\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.098784 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.098744 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:42.388707 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.388673 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr"] Sep 17 01:51:42.389672 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:51:42.389638 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce109529_b5bb_4f22_a47c_ab2defa1b772.slice/crio-7bd5c1fd83ff14ad1f2f747fc41863f2abd259b0ffb813a89c4900e9acbcb8a1 WatchSource:0}: Error finding container 7bd5c1fd83ff14ad1f2f747fc41863f2abd259b0ffb813a89c4900e9acbcb8a1: Status 404 returned error can't find the container with id 7bd5c1fd83ff14ad1f2f747fc41863f2abd259b0ffb813a89c4900e9acbcb8a1 Sep 17 01:51:42.773721 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.773678 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" event={"ID":"ce109529-b5bb-4f22-a47c-ab2defa1b772","Type":"ContainerStarted","Data":"db60cf65863d5f7d72423c64ff1b791da4ab07eadd7a5cbeecf389e865002b1d"} Sep 17 01:51:42.773889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.773752 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" event={"ID":"ce109529-b5bb-4f22-a47c-ab2defa1b772","Type":"ContainerStarted","Data":"7bd5c1fd83ff14ad1f2f747fc41863f2abd259b0ffb813a89c4900e9acbcb8a1"} Sep 17 01:51:42.791500 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:42.791432 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" podStartSLOduration=1.604196742 podStartE2EDuration="1.791413753s" totalImagesPullingTime="187.217011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:51:42.790707254 +0000 UTC m=+866.547474539" watchObservedRunningTime="2026-09-17 01:51:42.791413753 +0000 UTC m=+866.548180997" Sep 17 01:51:44.931633 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:44.931566 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:44.932138 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:44.931668 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:44.944447 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:44.944420 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:45.801413 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:45.801385 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-r2rmx" Sep 17 01:51:52.099361 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:52.099321 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:52.099872 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:52.099401 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:52.112104 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:52.112079 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:51:52.838036 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:51:52.838004 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bldfr" Sep 17 01:52:16.773059 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:52:16.772992 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:52:16.775388 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:52:16.775361 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:53:52.110603 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.110503 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68cbd6f44c-tb4zm"] Sep 17 01:53:52.114083 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.114066 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" Sep 17 01:53:52.121536 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.121456 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68cbd6f44c-tb4zm"] Sep 17 01:53:52.198203 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.198163 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/31aa68d9-e388-4cd2-99f2-71c02f71ade6-tls-cert\") pod \"authorino-68cbd6f44c-tb4zm\" (UID: \"31aa68d9-e388-4cd2-99f2-71c02f71ade6\") " pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" Sep 17 01:53:52.198358 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.198247 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c9pl\" (UniqueName: \"kubernetes.io/projected/31aa68d9-e388-4cd2-99f2-71c02f71ade6-kube-api-access-9c9pl\") pod \"authorino-68cbd6f44c-tb4zm\" (UID: \"31aa68d9-e388-4cd2-99f2-71c02f71ade6\") " pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" Sep 17 01:53:52.299341 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.299308 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c9pl\" (UniqueName: \"kubernetes.io/projected/31aa68d9-e388-4cd2-99f2-71c02f71ade6-kube-api-access-9c9pl\") pod \"authorino-68cbd6f44c-tb4zm\" (UID: \"31aa68d9-e388-4cd2-99f2-71c02f71ade6\") " pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" Sep 17 01:53:52.299502 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.299407 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/31aa68d9-e388-4cd2-99f2-71c02f71ade6-tls-cert\") pod \"authorino-68cbd6f44c-tb4zm\" (UID: \"31aa68d9-e388-4cd2-99f2-71c02f71ade6\") " pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" Sep 17 01:53:52.302054 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.302022 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/31aa68d9-e388-4cd2-99f2-71c02f71ade6-tls-cert\") pod \"authorino-68cbd6f44c-tb4zm\" (UID: \"31aa68d9-e388-4cd2-99f2-71c02f71ade6\") " pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" Sep 17 01:53:52.307164 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.307146 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c9pl\" (UniqueName: \"kubernetes.io/projected/31aa68d9-e388-4cd2-99f2-71c02f71ade6-kube-api-access-9c9pl\") pod \"authorino-68cbd6f44c-tb4zm\" (UID: \"31aa68d9-e388-4cd2-99f2-71c02f71ade6\") " pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" Sep 17 01:53:52.424147 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.424058 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" Sep 17 01:53:52.695077 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:52.695045 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68cbd6f44c-tb4zm"] Sep 17 01:53:52.696374 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:53:52.696347 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31aa68d9_e388_4cd2_99f2_71c02f71ade6.slice/crio-a999c76db6ebc705e548c3cd8dd6506772cdc3ac36ca1d16ee357c01621368c8 WatchSource:0}: Error finding container a999c76db6ebc705e548c3cd8dd6506772cdc3ac36ca1d16ee357c01621368c8: Status 404 returned error can't find the container with id a999c76db6ebc705e548c3cd8dd6506772cdc3ac36ca1d16ee357c01621368c8 Sep 17 01:53:53.519523 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:53.519480 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" event={"ID":"31aa68d9-e388-4cd2-99f2-71c02f71ade6","Type":"ContainerStarted","Data":"e8396d937a522ca024a4ca06850e31661db3e23e6630d43d3fa5b362192d126c"} Sep 17 01:53:53.520062 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:53.519535 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" event={"ID":"31aa68d9-e388-4cd2-99f2-71c02f71ade6","Type":"ContainerStarted","Data":"a999c76db6ebc705e548c3cd8dd6506772cdc3ac36ca1d16ee357c01621368c8"} Sep 17 01:53:53.537401 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:53.537332 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68cbd6f44c-tb4zm" podStartSLOduration=1.177124601 podStartE2EDuration="1.537310394s" totalImagesPullingTime="360.185793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:53:53.533428837 +0000 UTC m=+997.290196083" watchObservedRunningTime="2026-09-17 01:53:53.537310394 +0000 UTC m=+997.294077638" Sep 17 01:53:53.565810 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:53.565759 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dc9999c6b-2x2w7"] Sep 17 01:53:53.566543 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:53.566257 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" podUID="f8c05c0f-c60e-458c-baba-6ee6355ab0c3" containerName="authorino" containerID="cri-o://1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff" gracePeriod=30 Sep 17 01:53:53.838193 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:53.838168 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" Sep 17 01:53:53.915741 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:53.915710 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d87tp\" (UniqueName: \"kubernetes.io/projected/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-kube-api-access-d87tp\") pod \"f8c05c0f-c60e-458c-baba-6ee6355ab0c3\" (UID: \"f8c05c0f-c60e-458c-baba-6ee6355ab0c3\") " Sep 17 01:53:53.915884 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:53.915783 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-tls-cert\") pod \"f8c05c0f-c60e-458c-baba-6ee6355ab0c3\" (UID: \"f8c05c0f-c60e-458c-baba-6ee6355ab0c3\") " Sep 17 01:53:53.918025 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:53.917980 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-kube-api-access-d87tp" (OuterVolumeSpecName: "kube-api-access-d87tp") pod "f8c05c0f-c60e-458c-baba-6ee6355ab0c3" (UID: "f8c05c0f-c60e-458c-baba-6ee6355ab0c3"). InnerVolumeSpecName "kube-api-access-d87tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:53:53.927076 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:53.927042 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f8c05c0f-c60e-458c-baba-6ee6355ab0c3" (UID: "f8c05c0f-c60e-458c-baba-6ee6355ab0c3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:53:54.017477 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.017437 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d87tp\" (UniqueName: \"kubernetes.io/projected/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-kube-api-access-d87tp\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:53:54.017477 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.017480 2586 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8c05c0f-c60e-458c-baba-6ee6355ab0c3-tls-cert\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:53:54.042231 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.042181 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz"] Sep 17 01:53:54.042490 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.042442 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" podUID="e329c758-76f5-40a5-b5c1-86ff3a9d39c2" containerName="payload-pre-processing" containerID="cri-o://7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640" gracePeriod=30 Sep 17 01:53:54.313988 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.313965 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:53:54.422604 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.422509 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpjcd\" (UniqueName: \"kubernetes.io/projected/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-kube-api-access-qpjcd\") pod \"e329c758-76f5-40a5-b5c1-86ff3a9d39c2\" (UID: \"e329c758-76f5-40a5-b5c1-86ff3a9d39c2\") " Sep 17 01:53:54.422774 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.422640 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-plugins-config-volume\") pod \"e329c758-76f5-40a5-b5c1-86ff3a9d39c2\" (UID: \"e329c758-76f5-40a5-b5c1-86ff3a9d39c2\") " Sep 17 01:53:54.423000 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.422974 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e329c758-76f5-40a5-b5c1-86ff3a9d39c2" (UID: "e329c758-76f5-40a5-b5c1-86ff3a9d39c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:53:54.424969 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.424941 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-kube-api-access-qpjcd" (OuterVolumeSpecName: "kube-api-access-qpjcd") pod "e329c758-76f5-40a5-b5c1-86ff3a9d39c2" (UID: "e329c758-76f5-40a5-b5c1-86ff3a9d39c2"). InnerVolumeSpecName "kube-api-access-qpjcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:53:54.524000 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.523965 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-plugins-config-volume\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:53:54.524000 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.524000 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpjcd\" (UniqueName: \"kubernetes.io/projected/e329c758-76f5-40a5-b5c1-86ff3a9d39c2-kube-api-access-qpjcd\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:53:54.527912 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.527878 2586 generic.go:356] "Generic (PLEG): container finished" podID="f8c05c0f-c60e-458c-baba-6ee6355ab0c3" containerID="1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff" exitCode=0 Sep 17 01:53:54.528033 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.527927 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" Sep 17 01:53:54.528033 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.527967 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" event={"ID":"f8c05c0f-c60e-458c-baba-6ee6355ab0c3","Type":"ContainerDied","Data":"1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff"} Sep 17 01:53:54.528033 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.528014 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dc9999c6b-2x2w7" event={"ID":"f8c05c0f-c60e-458c-baba-6ee6355ab0c3","Type":"ContainerDied","Data":"73f8ed5adb87a752e4bf1e9a4ee57d06c9c62fc2c46e27841c97eaca818b51f1"} Sep 17 01:53:54.528197 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.528034 2586 scope.go:117] "RemoveContainer" containerID="1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff" Sep 17 01:53:54.529866 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.529838 2586 generic.go:356] "Generic (PLEG): container finished" podID="e329c758-76f5-40a5-b5c1-86ff3a9d39c2" containerID="7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640" exitCode=0 Sep 17 01:53:54.530001 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.529896 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" Sep 17 01:53:54.530001 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.529930 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" event={"ID":"e329c758-76f5-40a5-b5c1-86ff3a9d39c2","Type":"ContainerDied","Data":"7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640"} Sep 17 01:53:54.530001 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.529973 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz" event={"ID":"e329c758-76f5-40a5-b5c1-86ff3a9d39c2","Type":"ContainerDied","Data":"892cf0ea875d6dc59ede25d88c51c6b3dbf85255367b2a6277fda234f4c8bc0c"} Sep 17 01:53:54.540117 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.540097 2586 scope.go:117] "RemoveContainer" containerID="1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff" Sep 17 01:53:54.540457 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:53:54.540434 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff\": container with ID starting with 1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff not found: ID does not exist" containerID="1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff" Sep 17 01:53:54.540562 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.540461 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff"} err="failed to get container status \"1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff\": rpc error: code = NotFound desc = could not find container \"1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff\": container with ID starting with 1831ad7f8dede961342531d2c5362b9cd98db7001d61c4f1609f08cefaa666ff not found: ID does not exist" Sep 17 01:53:54.540562 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.540483 2586 scope.go:117] "RemoveContainer" containerID="7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640" Sep 17 01:53:54.550133 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.550108 2586 scope.go:117] "RemoveContainer" containerID="7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640" Sep 17 01:53:54.550700 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:53:54.550672 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640\": container with ID starting with 7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640 not found: ID does not exist" containerID="7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640" Sep 17 01:53:54.550829 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.550700 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640"} err="failed to get container status \"7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640\": rpc error: code = NotFound desc = could not find container \"7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640\": container with ID starting with 7034e2442ff5aec68a191045737c1583e37c9de5ad63371d1e201e8063758640 not found: ID does not exist" Sep 17 01:53:54.551121 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.551108 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dc9999c6b-2x2w7"] Sep 17 01:53:54.553614 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.553589 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7dc9999c6b-2x2w7"] Sep 17 01:53:54.562788 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.562749 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz"] Sep 17 01:53:54.565037 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.565015 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dxwzz"] Sep 17 01:53:54.768092 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.768015 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e329c758-76f5-40a5-b5c1-86ff3a9d39c2" path="/var/lib/kubelet/pods/e329c758-76f5-40a5-b5c1-86ff3a9d39c2/volumes" Sep 17 01:53:54.768357 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:53:54.768345 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c05c0f-c60e-458c-baba-6ee6355ab0c3" path="/var/lib/kubelet/pods/f8c05c0f-c60e-458c-baba-6ee6355ab0c3/volumes" Sep 17 01:54:08.519214 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.519178 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-88c7c5c76-xpj4g"] Sep 17 01:54:08.519721 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.519604 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c05c0f-c60e-458c-baba-6ee6355ab0c3" containerName="authorino" Sep 17 01:54:08.519721 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.519617 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c05c0f-c60e-458c-baba-6ee6355ab0c3" containerName="authorino" Sep 17 01:54:08.519721 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.519634 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e329c758-76f5-40a5-b5c1-86ff3a9d39c2" containerName="payload-pre-processing" Sep 17 01:54:08.519721 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.519639 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e329c758-76f5-40a5-b5c1-86ff3a9d39c2" containerName="payload-pre-processing" Sep 17 01:54:08.519721 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.519720 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c05c0f-c60e-458c-baba-6ee6355ab0c3" containerName="authorino" Sep 17 01:54:08.519889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.519732 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e329c758-76f5-40a5-b5c1-86ff3a9d39c2" containerName="payload-pre-processing" Sep 17 01:54:08.524394 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.524377 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" Sep 17 01:54:08.527290 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.527267 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-k4hf7\"" Sep 17 01:54:08.527532 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.527512 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Sep 17 01:54:08.529987 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.529960 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-88c7c5c76-xpj4g"] Sep 17 01:54:08.662800 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.662763 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrw2f\" (UniqueName: \"kubernetes.io/projected/6e4d0830-bc9e-470d-9930-f10f75fef77d-kube-api-access-nrw2f\") pod \"ai-gateway-controller-88c7c5c76-xpj4g\" (UID: \"6e4d0830-bc9e-470d-9930-f10f75fef77d\") " pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" Sep 17 01:54:08.763338 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.763305 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrw2f\" (UniqueName: \"kubernetes.io/projected/6e4d0830-bc9e-470d-9930-f10f75fef77d-kube-api-access-nrw2f\") pod \"ai-gateway-controller-88c7c5c76-xpj4g\" (UID: \"6e4d0830-bc9e-470d-9930-f10f75fef77d\") " pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" Sep 17 01:54:08.771202 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.771132 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrw2f\" (UniqueName: \"kubernetes.io/projected/6e4d0830-bc9e-470d-9930-f10f75fef77d-kube-api-access-nrw2f\") pod \"ai-gateway-controller-88c7c5c76-xpj4g\" (UID: \"6e4d0830-bc9e-470d-9930-f10f75fef77d\") " pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" Sep 17 01:54:08.835622 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:08.835566 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" Sep 17 01:54:09.112546 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:09.112506 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-88c7c5c76-xpj4g"] Sep 17 01:54:09.118007 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:54:09.117960 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4d0830_bc9e_470d_9930_f10f75fef77d.slice/crio-c423fad338b626f08016c8e26c22281b61001b718d3f809d574eaa9610999716 WatchSource:0}: Error finding container c423fad338b626f08016c8e26c22281b61001b718d3f809d574eaa9610999716: Status 404 returned error can't find the container with id c423fad338b626f08016c8e26c22281b61001b718d3f809d574eaa9610999716 Sep 17 01:54:09.119303 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:09.119286 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:54:09.615769 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:09.615738 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" event={"ID":"6e4d0830-bc9e-470d-9930-f10f75fef77d","Type":"ContainerStarted","Data":"c423fad338b626f08016c8e26c22281b61001b718d3f809d574eaa9610999716"} Sep 17 01:54:11.630634 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.630564 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" event={"ID":"6e4d0830-bc9e-470d-9930-f10f75fef77d","Type":"ContainerStarted","Data":"0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d"} Sep 17 01:54:11.631011 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.630663 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" Sep 17 01:54:11.649634 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.649569 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" podStartSLOduration=1.31396084 podStartE2EDuration="3.649555929s" totalImagesPullingTime="2.335595089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:54:11.648176844 +0000 UTC m=+1015.404944102" watchObservedRunningTime="2026-09-17 01:54:11.649555929 +0000 UTC m=+1015.406323173" Sep 17 01:54:11.938284 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.938252 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq"] Sep 17 01:54:11.941937 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.941921 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" Sep 17 01:54:11.944226 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.944207 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 17 01:54:11.952797 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.952773 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq"] Sep 17 01:54:11.957893 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.957863 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-gt5rn"] Sep 17 01:54:11.961361 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.961341 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" Sep 17 01:54:11.966797 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.966777 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4k2gd\"" Sep 17 01:54:11.976942 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:11.976919 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-gt5rn"] Sep 17 01:54:12.096757 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.096718 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d36f99-014a-474f-b1e0-74132bb0f4ac-plugins-config-volume\") pod \"payload-processing-85fd65c94c-gt5rn\" (UID: \"96d36f99-014a-474f-b1e0-74132bb0f4ac\") " pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" Sep 17 01:54:12.096915 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.096778 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f74e232-dbcf-4586-afbe-eba6a43eea00-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-ktpnq\" (UID: \"5f74e232-dbcf-4586-afbe-eba6a43eea00\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" Sep 17 01:54:12.097004 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.096979 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhj4\" (UniqueName: \"kubernetes.io/projected/5f74e232-dbcf-4586-afbe-eba6a43eea00-kube-api-access-vlhj4\") pod \"payload-pre-processing-76f657cdc4-ktpnq\" (UID: \"5f74e232-dbcf-4586-afbe-eba6a43eea00\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" Sep 17 01:54:12.097045 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.097026 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj4tf\" (UniqueName: \"kubernetes.io/projected/96d36f99-014a-474f-b1e0-74132bb0f4ac-kube-api-access-kj4tf\") pod \"payload-processing-85fd65c94c-gt5rn\" (UID: \"96d36f99-014a-474f-b1e0-74132bb0f4ac\") " pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" Sep 17 01:54:12.197881 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.197769 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f74e232-dbcf-4586-afbe-eba6a43eea00-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-ktpnq\" (UID: \"5f74e232-dbcf-4586-afbe-eba6a43eea00\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" Sep 17 01:54:12.197881 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.197874 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhj4\" (UniqueName: \"kubernetes.io/projected/5f74e232-dbcf-4586-afbe-eba6a43eea00-kube-api-access-vlhj4\") pod \"payload-pre-processing-76f657cdc4-ktpnq\" (UID: \"5f74e232-dbcf-4586-afbe-eba6a43eea00\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" Sep 17 01:54:12.198136 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.197897 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4tf\" (UniqueName: \"kubernetes.io/projected/96d36f99-014a-474f-b1e0-74132bb0f4ac-kube-api-access-kj4tf\") pod \"payload-processing-85fd65c94c-gt5rn\" (UID: \"96d36f99-014a-474f-b1e0-74132bb0f4ac\") " pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" Sep 17 01:54:12.198136 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.197937 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d36f99-014a-474f-b1e0-74132bb0f4ac-plugins-config-volume\") pod \"payload-processing-85fd65c94c-gt5rn\" (UID: \"96d36f99-014a-474f-b1e0-74132bb0f4ac\") " pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" Sep 17 01:54:12.198709 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.198688 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d36f99-014a-474f-b1e0-74132bb0f4ac-plugins-config-volume\") pod \"payload-processing-85fd65c94c-gt5rn\" (UID: \"96d36f99-014a-474f-b1e0-74132bb0f4ac\") " pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" Sep 17 01:54:12.198796 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.198720 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f74e232-dbcf-4586-afbe-eba6a43eea00-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-ktpnq\" (UID: \"5f74e232-dbcf-4586-afbe-eba6a43eea00\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" Sep 17 01:54:12.206158 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.206132 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhj4\" (UniqueName: \"kubernetes.io/projected/5f74e232-dbcf-4586-afbe-eba6a43eea00-kube-api-access-vlhj4\") pod \"payload-pre-processing-76f657cdc4-ktpnq\" (UID: \"5f74e232-dbcf-4586-afbe-eba6a43eea00\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" Sep 17 01:54:12.206257 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.206168 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj4tf\" (UniqueName: \"kubernetes.io/projected/96d36f99-014a-474f-b1e0-74132bb0f4ac-kube-api-access-kj4tf\") pod \"payload-processing-85fd65c94c-gt5rn\" (UID: \"96d36f99-014a-474f-b1e0-74132bb0f4ac\") " pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" Sep 17 01:54:12.253487 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.253457 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" Sep 17 01:54:12.271230 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.271206 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" Sep 17 01:54:12.530524 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.530476 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq"] Sep 17 01:54:12.530791 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:54:12.530770 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f74e232_dbcf_4586_afbe_eba6a43eea00.slice/crio-c082e8ac979c1cbbf3986fa4c2afeb09f3caf02e17d26257002dd5f46c0c19dc WatchSource:0}: Error finding container c082e8ac979c1cbbf3986fa4c2afeb09f3caf02e17d26257002dd5f46c0c19dc: Status 404 returned error can't find the container with id c082e8ac979c1cbbf3986fa4c2afeb09f3caf02e17d26257002dd5f46c0c19dc Sep 17 01:54:12.553198 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.553173 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-gt5rn"] Sep 17 01:54:12.554321 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:54:12.554297 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d36f99_014a_474f_b1e0_74132bb0f4ac.slice/crio-1fc793c4a2e3bc0b8dcf3e0585957356c5083320d2553aed26423adb5fc7ddd1 WatchSource:0}: Error finding container 1fc793c4a2e3bc0b8dcf3e0585957356c5083320d2553aed26423adb5fc7ddd1: Status 404 returned error can't find the container with id 1fc793c4a2e3bc0b8dcf3e0585957356c5083320d2553aed26423adb5fc7ddd1 Sep 17 01:54:12.637728 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.637696 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" event={"ID":"96d36f99-014a-474f-b1e0-74132bb0f4ac","Type":"ContainerStarted","Data":"1fc793c4a2e3bc0b8dcf3e0585957356c5083320d2553aed26423adb5fc7ddd1"} Sep 17 01:54:12.638856 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:12.638828 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" event={"ID":"5f74e232-dbcf-4586-afbe-eba6a43eea00","Type":"ContainerStarted","Data":"c082e8ac979c1cbbf3986fa4c2afeb09f3caf02e17d26257002dd5f46c0c19dc"} Sep 17 01:54:14.656308 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:14.656254 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" event={"ID":"96d36f99-014a-474f-b1e0-74132bb0f4ac","Type":"ContainerStarted","Data":"653c10949417167064fc615700ac5941127277a7e82908334588d033245204e1"} Sep 17 01:54:14.656804 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:14.656420 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" Sep 17 01:54:14.657836 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:14.657808 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" event={"ID":"5f74e232-dbcf-4586-afbe-eba6a43eea00","Type":"ContainerStarted","Data":"19e54cc434880749b545c73de41c806eb1a1d2df976b0a518ff862a2fa8d3f84"} Sep 17 01:54:14.657954 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:14.657935 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" Sep 17 01:54:14.673902 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:14.673856 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" podStartSLOduration=2.566450732 podStartE2EDuration="3.673841566s" totalImagesPullingTime="1.107390834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:54:14.67108031 +0000 UTC m=+1018.427847554" watchObservedRunningTime="2026-09-17 01:54:14.673841566 +0000 UTC m=+1018.430608807" Sep 17 01:54:14.685930 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:14.685889 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" podStartSLOduration=2.557039645 podStartE2EDuration="3.685875548s" totalImagesPullingTime="1.128835903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:54:14.684489609 +0000 UTC m=+1018.441256877" watchObservedRunningTime="2026-09-17 01:54:14.685875548 +0000 UTC m=+1018.442642790" Sep 17 01:54:22.641427 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:22.641393 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" Sep 17 01:54:25.666722 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:25.666692 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-85fd65c94c-gt5rn" Sep 17 01:54:25.667193 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:25.666774 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-76f657cdc4-ktpnq" Sep 17 01:54:36.417044 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.417001 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-88c7c5c76-xpj4g"] Sep 17 01:54:36.417563 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.417278 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" podUID="6e4d0830-bc9e-470d-9930-f10f75fef77d" containerName="manager" containerID="cri-o://0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d" gracePeriod=30 Sep 17 01:54:36.659932 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.659908 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" Sep 17 01:54:36.753615 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.753493 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrw2f\" (UniqueName: \"kubernetes.io/projected/6e4d0830-bc9e-470d-9930-f10f75fef77d-kube-api-access-nrw2f\") pod \"6e4d0830-bc9e-470d-9930-f10f75fef77d\" (UID: \"6e4d0830-bc9e-470d-9930-f10f75fef77d\") " Sep 17 01:54:36.755859 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.755828 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4d0830-bc9e-470d-9930-f10f75fef77d-kube-api-access-nrw2f" (OuterVolumeSpecName: "kube-api-access-nrw2f") pod "6e4d0830-bc9e-470d-9930-f10f75fef77d" (UID: "6e4d0830-bc9e-470d-9930-f10f75fef77d"). InnerVolumeSpecName "kube-api-access-nrw2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:54:36.790810 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.790776 2586 generic.go:356] "Generic (PLEG): container finished" podID="6e4d0830-bc9e-470d-9930-f10f75fef77d" containerID="0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d" exitCode=0 Sep 17 01:54:36.790991 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.790826 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" Sep 17 01:54:36.790991 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.790862 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" event={"ID":"6e4d0830-bc9e-470d-9930-f10f75fef77d","Type":"ContainerDied","Data":"0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d"} Sep 17 01:54:36.790991 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.790916 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-88c7c5c76-xpj4g" event={"ID":"6e4d0830-bc9e-470d-9930-f10f75fef77d","Type":"ContainerDied","Data":"c423fad338b626f08016c8e26c22281b61001b718d3f809d574eaa9610999716"} Sep 17 01:54:36.790991 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.790940 2586 scope.go:117] "RemoveContainer" containerID="0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d" Sep 17 01:54:36.799855 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.799834 2586 scope.go:117] "RemoveContainer" containerID="0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d" Sep 17 01:54:36.800118 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:54:36.800095 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d\": container with ID starting with 0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d not found: ID does not exist" containerID="0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d" Sep 17 01:54:36.800180 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.800128 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d"} err="failed to get container status \"0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d\": rpc error: code = NotFound desc = could not find container \"0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d\": container with ID starting with 0699f1af045983539633c6f910e922753a199c29e7f1f2af37f8d9d4511f6c1d not found: ID does not exist" Sep 17 01:54:36.806385 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.806363 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-88c7c5c76-xpj4g"] Sep 17 01:54:36.808820 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.808797 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-88c7c5c76-xpj4g"] Sep 17 01:54:36.854380 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:36.854354 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrw2f\" (UniqueName: \"kubernetes.io/projected/6e4d0830-bc9e-470d-9930-f10f75fef77d-kube-api-access-nrw2f\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:54:38.768805 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:54:38.768773 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e4d0830-bc9e-470d-9930-f10f75fef77d" path="/var/lib/kubelet/pods/6e4d0830-bc9e-470d-9930-f10f75fef77d/volumes" Sep 17 01:55:27.459719 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.459682 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd"] Sep 17 01:55:27.460398 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.460316 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4d0830-bc9e-470d-9930-f10f75fef77d" containerName="manager" Sep 17 01:55:27.460398 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.460336 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4d0830-bc9e-470d-9930-f10f75fef77d" containerName="manager" Sep 17 01:55:27.460560 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.460436 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4d0830-bc9e-470d-9930-f10f75fef77d" containerName="manager" Sep 17 01:55:27.465168 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.465148 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.467706 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.467686 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-38758b-openshift-default-dockercfg-2pwkz\"" Sep 17 01:55:27.473311 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.473292 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd"] Sep 17 01:55:27.537845 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.537791 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-socket\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.537845 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.537846 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-credential-socket\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.538078 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.537871 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-token\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.538078 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.537918 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-podinfo\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.538078 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.537943 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-certs\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.538078 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.537959 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-envoy\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.538258 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.538094 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istiod-ca-cert\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.538258 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.538160 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtd46\" (UniqueName: \"kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-kube-api-access-jtd46\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.538258 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.538245 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-data\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.639828 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.639790 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-data\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.639828 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.639830 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-socket\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640061 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.639848 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-credential-socket\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640061 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.639867 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-token\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640061 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.639895 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-podinfo\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640061 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.639916 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-certs\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640061 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.639940 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-envoy\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640061 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.639977 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istiod-ca-cert\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640368 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.640082 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtd46\" (UniqueName: \"kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-kube-api-access-jtd46\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640368 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.640241 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-data\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640368 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.640259 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-credential-socket\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640368 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.640320 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-socket\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640518 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.640363 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-certs\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.640825 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.640808 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istiod-ca-cert\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.642324 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.642303 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-envoy\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.642420 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.642407 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-podinfo\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.648687 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.648652 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtd46\" (UniqueName: \"kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-kube-api-access-jtd46\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.648817 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.648710 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-token\") pod \"e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:27.776504 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:27.776424 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:28.059645 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:28.059615 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd"] Sep 17 01:55:28.061125 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:55:28.061096 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6d4a6f_3f4c_4f52_a7c6_5dfa802d7766.slice/crio-ed4b20df9db4b1a6f9dafeb4aec3fd1a525178a7fa38884af50adb200e9aff23 WatchSource:0}: Error finding container ed4b20df9db4b1a6f9dafeb4aec3fd1a525178a7fa38884af50adb200e9aff23: Status 404 returned error can't find the container with id ed4b20df9db4b1a6f9dafeb4aec3fd1a525178a7fa38884af50adb200e9aff23 Sep 17 01:55:28.063527 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:28.063487 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:55:28.063625 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:28.063565 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:55:28.063625 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:28.063616 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:55:28.086926 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:28.086886 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" event={"ID":"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766","Type":"ContainerStarted","Data":"ed4b20df9db4b1a6f9dafeb4aec3fd1a525178a7fa38884af50adb200e9aff23"} Sep 17 01:55:29.095108 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:29.095072 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" event={"ID":"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766","Type":"ContainerStarted","Data":"59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448"} Sep 17 01:55:29.135302 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:29.135250 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" podStartSLOduration=2.135231439 podStartE2EDuration="2.135231439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:29.132165841 +0000 UTC m=+1092.888933126" watchObservedRunningTime="2026-09-17 01:55:29.135231439 +0000 UTC m=+1092.891998681" Sep 17 01:55:29.777044 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:29.777004 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:29.783479 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:29.783447 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:30.100984 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:30.100896 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:30.101875 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:30.101854 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:33.158745 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.158709 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh"] Sep 17 01:55:33.162718 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.162696 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 01:55:33.166749 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.166724 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-38758b-dockercfg-j8bkm\"" Sep 17 01:55:33.167176 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.167159 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-38758b\"" Sep 17 01:55:33.188140 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.188113 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh"] Sep 17 01:55:33.295542 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.295514 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a144920-e52a-4694-ab4f-bc5a612917db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh\" (UID: \"2a144920-e52a-4694-ab4f-bc5a612917db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 01:55:33.295743 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.295599 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6fg2\" (UniqueName: \"kubernetes.io/projected/2a144920-e52a-4694-ab4f-bc5a612917db-kube-api-access-r6fg2\") pod \"payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh\" (UID: \"2a144920-e52a-4694-ab4f-bc5a612917db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 01:55:33.396793 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.396760 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a144920-e52a-4694-ab4f-bc5a612917db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh\" (UID: \"2a144920-e52a-4694-ab4f-bc5a612917db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 01:55:33.396981 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.396822 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6fg2\" (UniqueName: \"kubernetes.io/projected/2a144920-e52a-4694-ab4f-bc5a612917db-kube-api-access-r6fg2\") pod \"payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh\" (UID: \"2a144920-e52a-4694-ab4f-bc5a612917db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 01:55:33.397520 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.397494 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a144920-e52a-4694-ab4f-bc5a612917db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh\" (UID: \"2a144920-e52a-4694-ab4f-bc5a612917db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 01:55:33.405351 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.405319 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6fg2\" (UniqueName: \"kubernetes.io/projected/2a144920-e52a-4694-ab4f-bc5a612917db-kube-api-access-r6fg2\") pod \"payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh\" (UID: \"2a144920-e52a-4694-ab4f-bc5a612917db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 01:55:33.473435 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.473352 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 01:55:33.754839 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:33.754807 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh"] Sep 17 01:55:33.756389 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:55:33.756360 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a144920_e52a_4694_ab4f_bc5a612917db.slice/crio-bef481beb7f3421b1e9311ce7001cba06f98c2725f9f0756255e0ef41fe919f5 WatchSource:0}: Error finding container bef481beb7f3421b1e9311ce7001cba06f98c2725f9f0756255e0ef41fe919f5: Status 404 returned error can't find the container with id bef481beb7f3421b1e9311ce7001cba06f98c2725f9f0756255e0ef41fe919f5 Sep 17 01:55:34.128340 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:34.128299 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" event={"ID":"2a144920-e52a-4694-ab4f-bc5a612917db","Type":"ContainerStarted","Data":"cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60"} Sep 17 01:55:34.128522 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:34.128362 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" event={"ID":"2a144920-e52a-4694-ab4f-bc5a612917db","Type":"ContainerStarted","Data":"bef481beb7f3421b1e9311ce7001cba06f98c2725f9f0756255e0ef41fe919f5"} Sep 17 01:55:34.128522 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:34.128401 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 01:55:34.145478 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:34.145423 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" podStartSLOduration=1.1454071080000001 podStartE2EDuration="1.145407108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:34.143065217 +0000 UTC m=+1097.899832478" watchObservedRunningTime="2026-09-17 01:55:34.145407108 +0000 UTC m=+1097.902174350" Sep 17 01:55:36.207346 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.207295 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd"] Sep 17 01:55:36.211130 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.211108 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 01:55:36.213543 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.213517 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e564e1\"" Sep 17 01:55:36.213755 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.213724 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e564e1-dockercfg-xsrdn\"" Sep 17 01:55:36.222434 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.222409 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd"] Sep 17 01:55:36.325430 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.325363 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2441bd-f353-48ee-ae3c-1ee3588d7f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd\" (UID: \"af2441bd-f353-48ee-ae3c-1ee3588d7f76\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 01:55:36.325713 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.325514 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2sw\" (UniqueName: \"kubernetes.io/projected/af2441bd-f353-48ee-ae3c-1ee3588d7f76-kube-api-access-qk2sw\") pod \"payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd\" (UID: \"af2441bd-f353-48ee-ae3c-1ee3588d7f76\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 01:55:36.426573 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.426534 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2441bd-f353-48ee-ae3c-1ee3588d7f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd\" (UID: \"af2441bd-f353-48ee-ae3c-1ee3588d7f76\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 01:55:36.426793 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.426699 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2sw\" (UniqueName: \"kubernetes.io/projected/af2441bd-f353-48ee-ae3c-1ee3588d7f76-kube-api-access-qk2sw\") pod \"payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd\" (UID: \"af2441bd-f353-48ee-ae3c-1ee3588d7f76\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 01:55:36.427438 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.427407 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2441bd-f353-48ee-ae3c-1ee3588d7f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd\" (UID: \"af2441bd-f353-48ee-ae3c-1ee3588d7f76\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 01:55:36.435317 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.435277 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2sw\" (UniqueName: \"kubernetes.io/projected/af2441bd-f353-48ee-ae3c-1ee3588d7f76-kube-api-access-qk2sw\") pod \"payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd\" (UID: \"af2441bd-f353-48ee-ae3c-1ee3588d7f76\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 01:55:36.523296 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.523200 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx"] Sep 17 01:55:36.523813 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.523776 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 01:55:36.529958 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.529936 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.534532 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.534508 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kpnw8\"" Sep 17 01:55:36.534949 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.534928 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-3f027f-serving-cert\"" Sep 17 01:55:36.535377 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.535358 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 01:55:36.535758 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.535726 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:55:36.536455 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.536413 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 01:55:36.539979 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.539953 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx"] Sep 17 01:55:36.641725 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.639160 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.641725 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.639272 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-metrics-certs\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.641725 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.639301 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ff8\" (UniqueName: \"kubernetes.io/projected/d2574ce8-de75-4639-9789-adbf5cc6c54d-kube-api-access-v2ff8\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.641725 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.639401 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2574ce8-de75-4639-9789-adbf5cc6c54d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.741610 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.741179 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-metrics-certs\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.741610 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.741234 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ff8\" (UniqueName: \"kubernetes.io/projected/d2574ce8-de75-4639-9789-adbf5cc6c54d-kube-api-access-v2ff8\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.741610 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.741327 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2574ce8-de75-4639-9789-adbf5cc6c54d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.741610 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.741375 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.741610 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:55:36.741516 2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-serving-cert: secret "maas-api-e2e-worker-w3-3f027f-serving-cert" not found Sep 17 01:55:36.742027 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:55:36.741633 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-maas-api-tls podName:d2574ce8-de75-4639-9789-adbf5cc6c54d nodeName:}" failed. No retries permitted until 2026-09-17 01:55:37.241611653 +0000 UTC m=+1100.998378876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-maas-api-tls") pod "maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" (UID: "d2574ce8-de75-4639-9789-adbf5cc6c54d") : secret "maas-api-e2e-worker-w3-3f027f-serving-cert" not found Sep 17 01:55:36.743004 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.742911 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2574ce8-de75-4639-9789-adbf5cc6c54d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.745364 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.745322 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-metrics-certs\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.754269 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.754239 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ff8\" (UniqueName: \"kubernetes.io/projected/d2574ce8-de75-4639-9789-adbf5cc6c54d-kube-api-access-v2ff8\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:36.846016 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:36.845717 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd"] Sep 17 01:55:36.847148 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:55:36.847103 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2441bd_f353_48ee_ae3c_1ee3588d7f76.slice/crio-5ce6bee48d744ed8c9bdb8ae6e87729a40510e41198e1ba5759774ce584718e3 WatchSource:0}: Error finding container 5ce6bee48d744ed8c9bdb8ae6e87729a40510e41198e1ba5759774ce584718e3: Status 404 returned error can't find the container with id 5ce6bee48d744ed8c9bdb8ae6e87729a40510e41198e1ba5759774ce584718e3 Sep 17 01:55:37.159473 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.159437 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" event={"ID":"af2441bd-f353-48ee-ae3c-1ee3588d7f76","Type":"ContainerStarted","Data":"00c7846d5146c838332492a19a1d414ae5eab6f802def29f00d67cd0cffd48e8"} Sep 17 01:55:37.159685 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.159504 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 01:55:37.159685 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.159527 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" event={"ID":"af2441bd-f353-48ee-ae3c-1ee3588d7f76","Type":"ContainerStarted","Data":"5ce6bee48d744ed8c9bdb8ae6e87729a40510e41198e1ba5759774ce584718e3"} Sep 17 01:55:37.190674 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.184527 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" podStartSLOduration=1.184506173 podStartE2EDuration="1.184506173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:37.183245832 +0000 UTC m=+1100.940013097" watchObservedRunningTime="2026-09-17 01:55:37.184506173 +0000 UTC m=+1100.941273418" Sep 17 01:55:37.255702 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.252393 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:37.259872 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.259249 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:37.358634 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.358567 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn"] Sep 17 01:55:37.362642 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.362624 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 01:55:37.365187 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.365155 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-3f027f-dockercfg-gk88b\"" Sep 17 01:55:37.365187 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.365175 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-3f027f\"" Sep 17 01:55:37.372328 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.372294 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn"] Sep 17 01:55:37.457752 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.457657 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cf525a-fc95-4c5c-a39c-16de265ba494-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn\" (UID: \"27cf525a-fc95-4c5c-a39c-16de265ba494\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 01:55:37.458118 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.458093 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7bf8\" (UniqueName: \"kubernetes.io/projected/27cf525a-fc95-4c5c-a39c-16de265ba494-kube-api-access-v7bf8\") pod \"payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn\" (UID: \"27cf525a-fc95-4c5c-a39c-16de265ba494\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 01:55:37.482382 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.482341 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:37.560226 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.560106 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cf525a-fc95-4c5c-a39c-16de265ba494-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn\" (UID: \"27cf525a-fc95-4c5c-a39c-16de265ba494\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 01:55:37.560441 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.560235 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7bf8\" (UniqueName: \"kubernetes.io/projected/27cf525a-fc95-4c5c-a39c-16de265ba494-kube-api-access-v7bf8\") pod \"payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn\" (UID: \"27cf525a-fc95-4c5c-a39c-16de265ba494\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 01:55:37.561230 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.561194 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cf525a-fc95-4c5c-a39c-16de265ba494-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn\" (UID: \"27cf525a-fc95-4c5c-a39c-16de265ba494\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 01:55:37.575063 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.573876 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7bf8\" (UniqueName: \"kubernetes.io/projected/27cf525a-fc95-4c5c-a39c-16de265ba494-kube-api-access-v7bf8\") pod \"payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn\" (UID: \"27cf525a-fc95-4c5c-a39c-16de265ba494\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 01:55:37.675264 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.675228 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 01:55:37.789791 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.789713 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx"] Sep 17 01:55:37.790773 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:55:37.790727 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2574ce8_de75_4639_9789_adbf5cc6c54d.slice/crio-7ad69aaa86e028a73bc910f0101dd5ddda32b36b30f59cde27871d2d3d6a7861 WatchSource:0}: Error finding container 7ad69aaa86e028a73bc910f0101dd5ddda32b36b30f59cde27871d2d3d6a7861: Status 404 returned error can't find the container with id 7ad69aaa86e028a73bc910f0101dd5ddda32b36b30f59cde27871d2d3d6a7861 Sep 17 01:55:37.976550 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:37.976519 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn"] Sep 17 01:55:37.977910 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:55:37.977884 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cf525a_fc95_4c5c_a39c_16de265ba494.slice/crio-e16e30872893547f11f9281051156c012eefc99634ec68c021f7c4b296b930ed WatchSource:0}: Error finding container e16e30872893547f11f9281051156c012eefc99634ec68c021f7c4b296b930ed: Status 404 returned error can't find the container with id e16e30872893547f11f9281051156c012eefc99634ec68c021f7c4b296b930ed Sep 17 01:55:38.168318 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:38.168274 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" event={"ID":"d2574ce8-de75-4639-9789-adbf5cc6c54d","Type":"ContainerStarted","Data":"7ad69aaa86e028a73bc910f0101dd5ddda32b36b30f59cde27871d2d3d6a7861"} Sep 17 01:55:38.170065 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:38.170040 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" event={"ID":"27cf525a-fc95-4c5c-a39c-16de265ba494","Type":"ContainerStarted","Data":"330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf"} Sep 17 01:55:38.170199 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:38.170116 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 01:55:38.170199 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:38.170160 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" event={"ID":"27cf525a-fc95-4c5c-a39c-16de265ba494","Type":"ContainerStarted","Data":"e16e30872893547f11f9281051156c012eefc99634ec68c021f7c4b296b930ed"} Sep 17 01:55:38.186821 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:38.186774 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" podStartSLOduration=1.186759139 podStartE2EDuration="1.186759139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:38.184149079 +0000 UTC m=+1101.940916341" watchObservedRunningTime="2026-09-17 01:55:38.186759139 +0000 UTC m=+1101.943526382" Sep 17 01:55:40.184032 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:40.183996 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" event={"ID":"d2574ce8-de75-4639-9789-adbf5cc6c54d","Type":"ContainerStarted","Data":"8f8a2dfffa1f3ee0808f07fba9d689fd3243173bf074e8eb64d7a223d62059cc"} Sep 17 01:55:40.184408 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:40.184052 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:40.204915 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:40.204859 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" podStartSLOduration=2.251641551 podStartE2EDuration="4.204835721s" totalImagesPullingTime="1.95319417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:40.201328016 +0000 UTC m=+1103.958095274" watchObservedRunningTime="2026-09-17 01:55:40.204835721 +0000 UTC m=+1103.961602964" Sep 17 01:55:45.140236 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:45.140204 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 01:55:45.168868 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:45.168721 2586 patch_prober.go:28] interesting pod/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": read tcp 10.132.0.2:41718->10.132.0.56:15021: read: connection reset by peer" start-of-body= Sep 17 01:55:45.168868 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:45.168786 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": read tcp 10.132.0.2:41718->10.132.0.56:15021: read: connection reset by peer" Sep 17 01:55:45.217033 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:45.217004 2586 generic.go:356] "Generic (PLEG): container finished" podID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerID="59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448" exitCode=0 Sep 17 01:55:45.217201 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:45.217077 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" event={"ID":"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766","Type":"ContainerDied","Data":"59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448"} Sep 17 01:55:45.217554 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:45.217538 2586 scope.go:117] "RemoveContainer" containerID="59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448" Sep 17 01:55:45.218733 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:45.218703 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:55:45.218813 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:45.218783 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:55:45.218855 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:45.218819 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:55:46.196892 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:46.196864 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 01:55:46.237470 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:46.237433 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" event={"ID":"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766","Type":"ContainerStarted","Data":"fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6"} Sep 17 01:55:46.777105 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:46.777070 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:46.777332 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:46.777311 2586 patch_prober.go:28] interesting pod/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 17 01:55:46.777395 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:46.777356 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 17 01:55:47.777837 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:47.777794 2586 patch_prober.go:28] interesting pod/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 17 01:55:47.778309 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:47.777858 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 17 01:55:48.172797 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:48.172761 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 01:55:48.777355 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:48.777317 2586 patch_prober.go:28] interesting pod/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 17 01:55:48.777562 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:48.777378 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 17 01:55:49.179545 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:49.179510 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 01:55:49.777013 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:49.776982 2586 patch_prober.go:28] interesting pod/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 17 01:55:49.777189 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:49.777037 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 17 01:55:50.365201 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.365169 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl"] Sep 17 01:55:50.371952 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.371923 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:55:50.379363 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.379341 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-266b61f5\"" Sep 17 01:55:50.393543 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.393518 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d921b751-6bfa-4303-afce-96247d349abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl\" (UID: \"d921b751-6bfa-4303-afce-96247d349abe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:55:50.393661 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.393599 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgq5s\" (UniqueName: \"kubernetes.io/projected/d921b751-6bfa-4303-afce-96247d349abe-kube-api-access-qgq5s\") pod \"payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl\" (UID: \"d921b751-6bfa-4303-afce-96247d349abe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:55:50.426364 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.426327 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl"] Sep 17 01:55:50.494338 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.494305 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d921b751-6bfa-4303-afce-96247d349abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl\" (UID: \"d921b751-6bfa-4303-afce-96247d349abe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:55:50.494530 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.494418 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgq5s\" (UniqueName: \"kubernetes.io/projected/d921b751-6bfa-4303-afce-96247d349abe-kube-api-access-qgq5s\") pod \"payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl\" (UID: \"d921b751-6bfa-4303-afce-96247d349abe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:55:50.495387 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.495365 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d921b751-6bfa-4303-afce-96247d349abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl\" (UID: \"d921b751-6bfa-4303-afce-96247d349abe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:55:50.518248 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.518225 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgq5s\" (UniqueName: \"kubernetes.io/projected/d921b751-6bfa-4303-afce-96247d349abe-kube-api-access-qgq5s\") pod \"payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl\" (UID: \"d921b751-6bfa-4303-afce-96247d349abe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:55:50.682222 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.682115 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:55:50.776996 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.776966 2586 patch_prober.go:28] interesting pod/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 17 01:55:50.777171 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.777016 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 17 01:55:50.995052 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:50.995010 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl"] Sep 17 01:55:50.996165 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:55:50.996127 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd921b751_6bfa_4303_afce_96247d349abe.slice/crio-09e5216bf79f8be2f1c4903ba41633af1635ac20973f6eb8b0cc4226193c5bc0 WatchSource:0}: Error finding container 09e5216bf79f8be2f1c4903ba41633af1635ac20973f6eb8b0cc4226193c5bc0: Status 404 returned error can't find the container with id 09e5216bf79f8be2f1c4903ba41633af1635ac20973f6eb8b0cc4226193c5bc0 Sep 17 01:55:51.279411 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.279320 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" event={"ID":"d921b751-6bfa-4303-afce-96247d349abe","Type":"ContainerStarted","Data":"a8a63f1e03e4a544d284f368c4d32ce28575ab50d1ca592ffb8f494471b98e5e"} Sep 17 01:55:51.279411 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.279379 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" event={"ID":"d921b751-6bfa-4303-afce-96247d349abe","Type":"ContainerStarted","Data":"09e5216bf79f8be2f1c4903ba41633af1635ac20973f6eb8b0cc4226193c5bc0"} Sep 17 01:55:51.279411 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.279404 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:55:51.301441 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.301388 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" podStartSLOduration=1.301371734 podStartE2EDuration="1.301371734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:51.300662311 +0000 UTC m=+1115.057429573" watchObservedRunningTime="2026-09-17 01:55:51.301371734 +0000 UTC m=+1115.058138976" Sep 17 01:55:51.780682 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.780651 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:51.781138 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.781099 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:51.781851 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.781834 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 01:55:51.810500 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.810467 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr"] Sep 17 01:55:51.814732 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.814711 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:51.817368 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.817350 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-55c648-openshift-default-dockercfg-srzpx\"" Sep 17 01:55:51.862745 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.862713 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr"] Sep 17 01:55:51.913513 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.913477 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-certs\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:51.913729 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.913534 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-data\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:51.913729 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.913627 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-socket\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:51.913729 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.913698 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-credential-socket\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:51.913887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.913733 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5249a534-f346-46eb-9e0e-e8c275efdf27-istiod-ca-cert\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:51.913887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.913797 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-token\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:51.913887 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.913839 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-podinfo\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:51.914011 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.913921 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj6hj\" (UniqueName: \"kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-kube-api-access-jj6hj\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:51.914011 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:51.913948 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-envoy\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.015541 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.015500 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-credential-socket\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.015761 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.015560 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5249a534-f346-46eb-9e0e-e8c275efdf27-istiod-ca-cert\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.015761 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.015656 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-token\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.015761 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.015692 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-podinfo\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.015945 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.015769 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj6hj\" (UniqueName: \"kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-kube-api-access-jj6hj\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.015945 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.015898 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-envoy\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.016050 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.015971 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-certs\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.016050 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.015998 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-data\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.016144 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.016054 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-socket\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.016144 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.016063 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-credential-socket\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.016360 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.016335 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5249a534-f346-46eb-9e0e-e8c275efdf27-istiod-ca-cert\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.016360 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.016360 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-data\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.016498 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.016428 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-socket\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.016741 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.016719 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-certs\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.018220 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.018198 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-envoy\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.018329 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.018313 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-podinfo\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.029298 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.029271 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-token\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.030465 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.030442 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj6hj\" (UniqueName: \"kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-kube-api-access-jj6hj\") pod \"e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.126316 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.126280 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:52.426892 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.426866 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr"] Sep 17 01:55:52.428466 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:55:52.428443 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5249a534_f346_46eb_9e0e_e8c275efdf27.slice/crio-607e2731c0f1fcf76c551668a4ee1c5ed429843518111af9ba05be74fa0c0c2a WatchSource:0}: Error finding container 607e2731c0f1fcf76c551668a4ee1c5ed429843518111af9ba05be74fa0c0c2a: Status 404 returned error can't find the container with id 607e2731c0f1fcf76c551668a4ee1c5ed429843518111af9ba05be74fa0c0c2a Sep 17 01:55:52.431121 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.431091 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:55:52.431199 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.431155 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:55:52.431199 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:52.431190 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:55:53.292880 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:53.292840 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" event={"ID":"5249a534-f346-46eb-9e0e-e8c275efdf27","Type":"ContainerStarted","Data":"105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3"} Sep 17 01:55:53.293270 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:53.292907 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" event={"ID":"5249a534-f346-46eb-9e0e-e8c275efdf27","Type":"ContainerStarted","Data":"607e2731c0f1fcf76c551668a4ee1c5ed429843518111af9ba05be74fa0c0c2a"} Sep 17 01:55:53.320961 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:53.320906 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" podStartSLOduration=2.320885911 podStartE2EDuration="2.320885911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:53.320601406 +0000 UTC m=+1117.077368671" watchObservedRunningTime="2026-09-17 01:55:53.320885911 +0000 UTC m=+1117.077653155" Sep 17 01:55:54.126736 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:54.126694 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:54.132485 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:54.132455 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:54.299344 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:54.299308 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:54.300214 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:54.300197 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:55:59.639275 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.639239 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6"] Sep 17 01:55:59.644064 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.644042 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.647101 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.647070 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-55c648-serving-cert\"" Sep 17 01:55:59.660466 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.660438 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6"] Sep 17 01:55:59.703476 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.703441 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-maas-api-tls\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.703661 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.703515 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7482277-e9b3-4fd5-8d8d-73293a42c40c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.703661 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.703556 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-metrics-certs\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.703763 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.703688 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l226p\" (UniqueName: \"kubernetes.io/projected/f7482277-e9b3-4fd5-8d8d-73293a42c40c-kube-api-access-l226p\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.804856 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.804824 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l226p\" (UniqueName: \"kubernetes.io/projected/f7482277-e9b3-4fd5-8d8d-73293a42c40c-kube-api-access-l226p\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.805022 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.804902 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-maas-api-tls\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.805022 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.804986 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7482277-e9b3-4fd5-8d8d-73293a42c40c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.805105 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.805035 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-metrics-certs\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.805105 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:55:59.805050 2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-serving-cert: secret "maas-api-e2e-inf-b-55c648-serving-cert" not found Sep 17 01:55:59.805190 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:55:59.805133 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-maas-api-tls podName:f7482277-e9b3-4fd5-8d8d-73293a42c40c nodeName:}" failed. No retries permitted until 2026-09-17 01:56:00.305112372 +0000 UTC m=+1124.061879593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-maas-api-tls") pod "maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" (UID: "f7482277-e9b3-4fd5-8d8d-73293a42c40c") : secret "maas-api-e2e-inf-b-55c648-serving-cert" not found Sep 17 01:55:59.805742 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.805726 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7482277-e9b3-4fd5-8d8d-73293a42c40c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.807500 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.807483 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-metrics-certs\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:55:59.814925 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:55:59.814900 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l226p\" (UniqueName: \"kubernetes.io/projected/f7482277-e9b3-4fd5-8d8d-73293a42c40c-kube-api-access-l226p\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:56:00.311349 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:00.311311 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-maas-api-tls\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:56:00.313889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:00.313870 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-maas-api-tls\") pod \"maas-api-e2e-inf-b-55c648-59496859cc-8tqr6\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:56:00.556992 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:00.556960 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:56:00.839845 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:00.839815 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6"] Sep 17 01:56:00.840367 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:56:00.840338 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7482277_e9b3_4fd5_8d8d_73293a42c40c.slice/crio-c7b31c87e74ee5be405af55b00fd7e1a25fe6d51134041aeb6790d451164cf77 WatchSource:0}: Error finding container c7b31c87e74ee5be405af55b00fd7e1a25fe6d51134041aeb6790d451164cf77: Status 404 returned error can't find the container with id c7b31c87e74ee5be405af55b00fd7e1a25fe6d51134041aeb6790d451164cf77 Sep 17 01:56:01.343411 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:01.343372 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" event={"ID":"f7482277-e9b3-4fd5-8d8d-73293a42c40c","Type":"ContainerStarted","Data":"842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef"} Sep 17 01:56:01.343620 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:01.343440 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" event={"ID":"f7482277-e9b3-4fd5-8d8d-73293a42c40c","Type":"ContainerStarted","Data":"c7b31c87e74ee5be405af55b00fd7e1a25fe6d51134041aeb6790d451164cf77"} Sep 17 01:56:01.343620 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:01.343566 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:56:01.374334 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:01.374279 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" podStartSLOduration=1.977217751 podStartE2EDuration="2.37426221s" totalImagesPullingTime="397.044459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:56:01.372712734 +0000 UTC m=+1125.129480016" watchObservedRunningTime="2026-09-17 01:56:01.37426221 +0000 UTC m=+1125.131029487" Sep 17 01:56:02.288126 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:02.288093 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:56:07.355967 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:07.355927 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 01:56:10.269549 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.269515 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl"] Sep 17 01:56:10.270062 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.269797 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" podUID="d921b751-6bfa-4303-afce-96247d349abe" containerName="payload-pre-processing" containerID="cri-o://a8a63f1e03e4a544d284f368c4d32ce28575ab50d1ca592ffb8f494471b98e5e" gracePeriod=30 Sep 17 01:56:10.404030 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.403980 2586 generic.go:356] "Generic (PLEG): container finished" podID="d921b751-6bfa-4303-afce-96247d349abe" containerID="a8a63f1e03e4a544d284f368c4d32ce28575ab50d1ca592ffb8f494471b98e5e" exitCode=0 Sep 17 01:56:10.404189 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.404053 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" event={"ID":"d921b751-6bfa-4303-afce-96247d349abe","Type":"ContainerDied","Data":"a8a63f1e03e4a544d284f368c4d32ce28575ab50d1ca592ffb8f494471b98e5e"} Sep 17 01:56:10.584865 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.584833 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:56:10.731606 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.730675 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgq5s\" (UniqueName: \"kubernetes.io/projected/d921b751-6bfa-4303-afce-96247d349abe-kube-api-access-qgq5s\") pod \"d921b751-6bfa-4303-afce-96247d349abe\" (UID: \"d921b751-6bfa-4303-afce-96247d349abe\") " Sep 17 01:56:10.731606 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.730913 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d921b751-6bfa-4303-afce-96247d349abe-plugins-config-volume\") pod \"d921b751-6bfa-4303-afce-96247d349abe\" (UID: \"d921b751-6bfa-4303-afce-96247d349abe\") " Sep 17 01:56:10.731873 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.731768 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d921b751-6bfa-4303-afce-96247d349abe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d921b751-6bfa-4303-afce-96247d349abe" (UID: "d921b751-6bfa-4303-afce-96247d349abe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:56:10.744857 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.744813 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d921b751-6bfa-4303-afce-96247d349abe-kube-api-access-qgq5s" (OuterVolumeSpecName: "kube-api-access-qgq5s") pod "d921b751-6bfa-4303-afce-96247d349abe" (UID: "d921b751-6bfa-4303-afce-96247d349abe"). InnerVolumeSpecName "kube-api-access-qgq5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:56:10.835183 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.835072 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgq5s\" (UniqueName: \"kubernetes.io/projected/d921b751-6bfa-4303-afce-96247d349abe-kube-api-access-qgq5s\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:56:10.835445 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:10.835427 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d921b751-6bfa-4303-afce-96247d349abe-plugins-config-volume\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:56:11.411532 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:11.411502 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" Sep 17 01:56:11.411532 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:11.411514 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl" event={"ID":"d921b751-6bfa-4303-afce-96247d349abe","Type":"ContainerDied","Data":"09e5216bf79f8be2f1c4903ba41633af1635ac20973f6eb8b0cc4226193c5bc0"} Sep 17 01:56:11.412064 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:11.411570 2586 scope.go:117] "RemoveContainer" containerID="a8a63f1e03e4a544d284f368c4d32ce28575ab50d1ca592ffb8f494471b98e5e" Sep 17 01:56:11.434944 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:11.434884 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl"] Sep 17 01:56:11.440654 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:11.440604 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-266b61f5-c7c7d5fb6-7vsdl"] Sep 17 01:56:12.768432 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:12.768397 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d921b751-6bfa-4303-afce-96247d349abe" path="/var/lib/kubelet/pods/d921b751-6bfa-4303-afce-96247d349abe/volumes" Sep 17 01:56:17.645844 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.645802 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs"] Sep 17 01:56:17.646536 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.646510 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d921b751-6bfa-4303-afce-96247d349abe" containerName="payload-pre-processing" Sep 17 01:56:17.646658 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.646539 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d921b751-6bfa-4303-afce-96247d349abe" containerName="payload-pre-processing" Sep 17 01:56:17.646718 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.646695 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d921b751-6bfa-4303-afce-96247d349abe" containerName="payload-pre-processing" Sep 17 01:56:17.651734 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.651706 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.655552 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.655474 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-404126c3-serving-cert\"" Sep 17 01:56:17.665170 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.665134 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs"] Sep 17 01:56:17.714460 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.714421 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crsc\" (UniqueName: \"kubernetes.io/projected/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-kube-api-access-6crsc\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.714664 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.714495 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-maas-api-tls\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.714664 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.714598 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.714664 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.714622 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-metrics-certs\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.815360 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.815318 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.815632 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.815368 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-metrics-certs\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.815632 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.815481 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crsc\" (UniqueName: \"kubernetes.io/projected/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-kube-api-access-6crsc\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.815790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.815629 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-maas-api-tls\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.815864 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:56:17.815807 2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-serving-cert: secret "maas-api-e2e-migrate-404126c3-serving-cert" not found Sep 17 01:56:17.815971 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:56:17.815952 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-maas-api-tls podName:8a2f7e4a-597c-47ac-af5e-137e5dba04c9 nodeName:}" failed. No retries permitted until 2026-09-17 01:56:18.315930466 +0000 UTC m=+1142.072697702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-maas-api-tls") pod "maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" (UID: "8a2f7e4a-597c-47ac-af5e-137e5dba04c9") : secret "maas-api-e2e-migrate-404126c3-serving-cert" not found Sep 17 01:56:17.816155 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.816125 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.818520 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.818499 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-metrics-certs\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:17.826884 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:17.826854 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crsc\" (UniqueName: \"kubernetes.io/projected/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-kube-api-access-6crsc\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:18.322330 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:18.322290 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-maas-api-tls\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:18.325023 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:18.324989 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-maas-api-tls\") pod \"maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:18.567107 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:18.567060 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:18.875020 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:56:18.874976 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2f7e4a_597c_47ac_af5e_137e5dba04c9.slice/crio-4cd7094cf9e7bd217237c32391953e09517b3baccd9e2f28e6dfa55d3d572178 WatchSource:0}: Error finding container 4cd7094cf9e7bd217237c32391953e09517b3baccd9e2f28e6dfa55d3d572178: Status 404 returned error can't find the container with id 4cd7094cf9e7bd217237c32391953e09517b3baccd9e2f28e6dfa55d3d572178 Sep 17 01:56:18.877523 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:18.877495 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs"] Sep 17 01:56:19.482892 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:19.482853 2586 generic.go:356] "Generic (PLEG): container finished" podID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerID="105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3" exitCode=0 Sep 17 01:56:19.483144 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:19.482924 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" event={"ID":"5249a534-f346-46eb-9e0e-e8c275efdf27","Type":"ContainerDied","Data":"105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3"} Sep 17 01:56:19.483477 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:19.483456 2586 scope.go:117] "RemoveContainer" containerID="105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3" Sep 17 01:56:19.484947 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:19.484920 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" event={"ID":"8a2f7e4a-597c-47ac-af5e-137e5dba04c9","Type":"ContainerStarted","Data":"629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef"} Sep 17 01:56:19.485046 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:19.484940 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:56:19.485046 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:19.484984 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:19.485046 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:19.485009 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" event={"ID":"8a2f7e4a-597c-47ac-af5e-137e5dba04c9","Type":"ContainerStarted","Data":"4cd7094cf9e7bd217237c32391953e09517b3baccd9e2f28e6dfa55d3d572178"} Sep 17 01:56:19.485173 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:19.485028 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:56:19.485227 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:19.485156 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 17 01:56:19.523078 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:19.518919 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" podStartSLOduration=2.202583886 podStartE2EDuration="2.518896559s" totalImagesPullingTime="316.312673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:56:19.518368709 +0000 UTC m=+1143.275135970" watchObservedRunningTime="2026-09-17 01:56:19.518896559 +0000 UTC m=+1143.275663804" Sep 17 01:56:20.493835 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:20.493798 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" event={"ID":"5249a534-f346-46eb-9e0e-e8c275efdf27","Type":"ContainerStarted","Data":"dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a"} Sep 17 01:56:21.127161 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:21.127119 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:56:21.128230 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:21.128202 2586 patch_prober.go:28] interesting pod/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 17 01:56:21.128450 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:21.128425 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 17 01:56:22.127678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:22.127619 2586 patch_prober.go:28] interesting pod/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 17 01:56:22.128199 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:22.127705 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 17 01:56:23.126959 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:23.126921 2586 patch_prober.go:28] interesting pod/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 17 01:56:23.127283 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:23.127244 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 17 01:56:24.127243 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:24.127194 2586 patch_prober.go:28] interesting pod/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 17 01:56:24.127794 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:24.127290 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 17 01:56:24.300018 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:24.299960 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:56:25.127192 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:25.127143 2586 patch_prober.go:28] interesting pod/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 17 01:56:25.127432 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:25.127217 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 17 01:56:25.500350 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:25.500275 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:26.132575 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:26.132542 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:56:26.133678 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:26.133657 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 01:56:56.123243 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.123147 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs"] Sep 17 01:56:56.124174 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.123537 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" podUID="8a2f7e4a-597c-47ac-af5e-137e5dba04c9" containerName="maas-api" containerID="cri-o://629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef" gracePeriod=30 Sep 17 01:56:56.383031 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.382955 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:56.438419 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.438386 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-metrics-certs\") pod \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " Sep 17 01:56:56.438665 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.438456 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6crsc\" (UniqueName: \"kubernetes.io/projected/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-kube-api-access-6crsc\") pod \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " Sep 17 01:56:56.438665 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.438493 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-maas-api-tls\") pod \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " Sep 17 01:56:56.438665 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.438522 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-openshift-service-ca\") pod \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\" (UID: \"8a2f7e4a-597c-47ac-af5e-137e5dba04c9\") " Sep 17 01:56:56.445607 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.445446 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a2f7e4a-597c-47ac-af5e-137e5dba04c9" (UID: "8a2f7e4a-597c-47ac-af5e-137e5dba04c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:56:56.446871 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.446709 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a2f7e4a-597c-47ac-af5e-137e5dba04c9" (UID: "8a2f7e4a-597c-47ac-af5e-137e5dba04c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:56:56.448242 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.448201 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a2f7e4a-597c-47ac-af5e-137e5dba04c9" (UID: "8a2f7e4a-597c-47ac-af5e-137e5dba04c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:56:56.451906 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.451869 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-kube-api-access-6crsc" (OuterVolumeSpecName: "kube-api-access-6crsc") pod "8a2f7e4a-597c-47ac-af5e-137e5dba04c9" (UID: "8a2f7e4a-597c-47ac-af5e-137e5dba04c9"). InnerVolumeSpecName "kube-api-access-6crsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:56:56.539730 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.539689 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:56:56.539730 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.539730 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6crsc\" (UniqueName: \"kubernetes.io/projected/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-kube-api-access-6crsc\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:56:56.539957 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.539745 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:56:56.539957 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.539757 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a2f7e4a-597c-47ac-af5e-137e5dba04c9-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:56:56.735647 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.735544 2586 generic.go:356] "Generic (PLEG): container finished" podID="8a2f7e4a-597c-47ac-af5e-137e5dba04c9" containerID="629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef" exitCode=0 Sep 17 01:56:56.735647 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.735602 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" Sep 17 01:56:56.735647 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.735638 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" event={"ID":"8a2f7e4a-597c-47ac-af5e-137e5dba04c9","Type":"ContainerDied","Data":"629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef"} Sep 17 01:56:56.735889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.735693 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs" event={"ID":"8a2f7e4a-597c-47ac-af5e-137e5dba04c9","Type":"ContainerDied","Data":"4cd7094cf9e7bd217237c32391953e09517b3baccd9e2f28e6dfa55d3d572178"} Sep 17 01:56:56.735889 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.735720 2586 scope.go:117] "RemoveContainer" containerID="629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef" Sep 17 01:56:56.754500 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.754477 2586 scope.go:117] "RemoveContainer" containerID="629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef" Sep 17 01:56:56.754913 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:56:56.754880 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef\": container with ID starting with 629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef not found: ID does not exist" containerID="629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef" Sep 17 01:56:56.755009 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.754918 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef"} err="failed to get container status \"629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef\": rpc error: code = NotFound desc = could not find container \"629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef\": container with ID starting with 629d5ddfac310c9006dca7fe7717585f5e5b2f0ac1735aa2a889708529f4f0ef not found: ID does not exist" Sep 17 01:56:56.769410 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.769381 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs"] Sep 17 01:56:56.769410 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:56.769419 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-404126c3-8455b88b76-xkqhs"] Sep 17 01:56:58.768651 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:56:58.768606 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a2f7e4a-597c-47ac-af5e-137e5dba04c9" path="/var/lib/kubelet/pods/8a2f7e4a-597c-47ac-af5e-137e5dba04c9/volumes" Sep 17 01:57:03.616692 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.616657 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq"] Sep 17 01:57:03.617132 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.617119 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a2f7e4a-597c-47ac-af5e-137e5dba04c9" containerName="maas-api" Sep 17 01:57:03.617177 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.617134 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a2f7e4a-597c-47ac-af5e-137e5dba04c9" containerName="maas-api" Sep 17 01:57:03.617243 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.617234 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a2f7e4a-597c-47ac-af5e-137e5dba04c9" containerName="maas-api" Sep 17 01:57:03.622076 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.622052 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.624495 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.624473 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3607043f-serving-cert\"" Sep 17 01:57:03.631348 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.631319 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq"] Sep 17 01:57:03.714997 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.714958 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-maas-api-tls\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.715168 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.715066 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-openshift-service-ca\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.715168 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.715102 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc7ch\" (UniqueName: \"kubernetes.io/projected/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-kube-api-access-wc7ch\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.715168 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.715121 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-metrics-certs\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.815776 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.815738 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-maas-api-tls\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.815992 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.815828 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-openshift-service-ca\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.815992 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.815863 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc7ch\" (UniqueName: \"kubernetes.io/projected/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-kube-api-access-wc7ch\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.815992 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.815882 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-metrics-certs\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.816503 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.816481 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-openshift-service-ca\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.818412 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.818390 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-maas-api-tls\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.818507 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.818468 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-metrics-certs\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.823906 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.823875 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc7ch\" (UniqueName: \"kubernetes.io/projected/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-kube-api-access-wc7ch\") pod \"maas-api-e2e-ait-3607043f-6785b765d9-jrcsq\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:03.934650 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:03.934548 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:04.216737 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:04.216698 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq"] Sep 17 01:57:04.218220 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:57:04.218192 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b2a5b3_0330_4dfd_ab46_a42eaa7046c3.slice/crio-4239a2655c37fb4d53a1dfbc1bf11361a4fc488d9d756d30a69df0f355dcb21c WatchSource:0}: Error finding container 4239a2655c37fb4d53a1dfbc1bf11361a4fc488d9d756d30a69df0f355dcb21c: Status 404 returned error can't find the container with id 4239a2655c37fb4d53a1dfbc1bf11361a4fc488d9d756d30a69df0f355dcb21c Sep 17 01:57:04.786620 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:04.786560 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" event={"ID":"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3","Type":"ContainerStarted","Data":"2e10cc06780bf71e5b7526a7a73365e9378e16cdcc0fe61f7a8c0cfed886a1e2"} Sep 17 01:57:04.787010 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:04.786634 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" event={"ID":"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3","Type":"ContainerStarted","Data":"4239a2655c37fb4d53a1dfbc1bf11361a4fc488d9d756d30a69df0f355dcb21c"} Sep 17 01:57:04.787010 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:04.786747 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:04.803397 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:04.803327 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" podStartSLOduration=1.419547082 podStartE2EDuration="1.803306572s" totalImagesPullingTime="383.75949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:57:04.802985187 +0000 UTC m=+1188.559752452" watchObservedRunningTime="2026-09-17 01:57:04.803306572 +0000 UTC m=+1188.560073819" Sep 17 01:57:10.799408 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:10.799367 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:16.826414 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:16.826372 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:57:16.830856 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:16.830831 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 01:57:41.883192 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:41.883156 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq"] Sep 17 01:57:41.883747 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:41.883417 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" podUID="71b2a5b3-0330-4dfd-ab46-a42eaa7046c3" containerName="maas-api" containerID="cri-o://2e10cc06780bf71e5b7526a7a73365e9378e16cdcc0fe61f7a8c0cfed886a1e2" gracePeriod=30 Sep 17 01:57:42.022531 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.022497 2586 generic.go:356] "Generic (PLEG): container finished" podID="71b2a5b3-0330-4dfd-ab46-a42eaa7046c3" containerID="2e10cc06780bf71e5b7526a7a73365e9378e16cdcc0fe61f7a8c0cfed886a1e2" exitCode=0 Sep 17 01:57:42.022713 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.022572 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" event={"ID":"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3","Type":"ContainerDied","Data":"2e10cc06780bf71e5b7526a7a73365e9378e16cdcc0fe61f7a8c0cfed886a1e2"} Sep 17 01:57:42.134422 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.134364 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:42.226401 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.226368 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-openshift-service-ca\") pod \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " Sep 17 01:57:42.226573 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.226418 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-metrics-certs\") pod \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " Sep 17 01:57:42.226573 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.226452 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-maas-api-tls\") pod \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " Sep 17 01:57:42.226573 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.226518 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc7ch\" (UniqueName: \"kubernetes.io/projected/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-kube-api-access-wc7ch\") pod \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\" (UID: \"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3\") " Sep 17 01:57:42.226800 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.226747 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71b2a5b3-0330-4dfd-ab46-a42eaa7046c3" (UID: "71b2a5b3-0330-4dfd-ab46-a42eaa7046c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:57:42.226948 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.226926 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:57:42.228799 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.228773 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71b2a5b3-0330-4dfd-ab46-a42eaa7046c3" (UID: "71b2a5b3-0330-4dfd-ab46-a42eaa7046c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:57:42.228799 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.228787 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71b2a5b3-0330-4dfd-ab46-a42eaa7046c3" (UID: "71b2a5b3-0330-4dfd-ab46-a42eaa7046c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:57:42.228951 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.228782 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-kube-api-access-wc7ch" (OuterVolumeSpecName: "kube-api-access-wc7ch") pod "71b2a5b3-0330-4dfd-ab46-a42eaa7046c3" (UID: "71b2a5b3-0330-4dfd-ab46-a42eaa7046c3"). InnerVolumeSpecName "kube-api-access-wc7ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:57:42.328170 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.328139 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc7ch\" (UniqueName: \"kubernetes.io/projected/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-kube-api-access-wc7ch\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:57:42.328170 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.328169 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:57:42.328372 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:42.328179 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:57:43.030796 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:43.030701 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" event={"ID":"71b2a5b3-0330-4dfd-ab46-a42eaa7046c3","Type":"ContainerDied","Data":"4239a2655c37fb4d53a1dfbc1bf11361a4fc488d9d756d30a69df0f355dcb21c"} Sep 17 01:57:43.030796 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:43.030763 2586 scope.go:117] "RemoveContainer" containerID="2e10cc06780bf71e5b7526a7a73365e9378e16cdcc0fe61f7a8c0cfed886a1e2" Sep 17 01:57:43.031256 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:43.030719 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq" Sep 17 01:57:43.050141 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:43.050107 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq"] Sep 17 01:57:43.054230 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:43.054202 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3607043f-6785b765d9-jrcsq"] Sep 17 01:57:44.768196 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:44.768158 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b2a5b3-0330-4dfd-ab46-a42eaa7046c3" path="/var/lib/kubelet/pods/71b2a5b3-0330-4dfd-ab46-a42eaa7046c3/volumes" Sep 17 01:57:50.227564 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.227525 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz"] Sep 17 01:57:50.228261 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.228243 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b2a5b3-0330-4dfd-ab46-a42eaa7046c3" containerName="maas-api" Sep 17 01:57:50.228346 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.228264 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b2a5b3-0330-4dfd-ab46-a42eaa7046c3" containerName="maas-api" Sep 17 01:57:50.228425 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.228414 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b2a5b3-0330-4dfd-ab46-a42eaa7046c3" containerName="maas-api" Sep 17 01:57:50.232350 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.232331 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.234858 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.234837 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-56574e4f-serving-cert\"" Sep 17 01:57:50.244408 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.244378 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz"] Sep 17 01:57:50.305168 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.305140 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-metrics-certs\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.305320 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.305216 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h44b\" (UniqueName: \"kubernetes.io/projected/ab66b979-d535-4d31-bc5d-d0693cdc572d-kube-api-access-6h44b\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.305320 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.305312 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab66b979-d535-4d31-bc5d-d0693cdc572d-openshift-service-ca\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.305403 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.305356 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-maas-api-tls\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.406073 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.406037 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab66b979-d535-4d31-bc5d-d0693cdc572d-openshift-service-ca\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.406242 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.406082 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-maas-api-tls\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.406242 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.406115 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-metrics-certs\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.406242 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.406148 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h44b\" (UniqueName: \"kubernetes.io/projected/ab66b979-d535-4d31-bc5d-d0693cdc572d-kube-api-access-6h44b\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.406848 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.406813 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab66b979-d535-4d31-bc5d-d0693cdc572d-openshift-service-ca\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.408671 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.408645 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-metrics-certs\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.408770 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.408749 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-maas-api-tls\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.414013 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.413987 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h44b\" (UniqueName: \"kubernetes.io/projected/ab66b979-d535-4d31-bc5d-d0693cdc572d-kube-api-access-6h44b\") pod \"maas-api-e2e-derive-56574e4f-8694488759-mzsjz\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.544864 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.544771 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:50.820891 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:50.820864 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz"] Sep 17 01:57:50.822986 ip-10-0-143-174 kubenswrapper[2586]: W0917 01:57:50.822958 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab66b979_d535_4d31_bc5d_d0693cdc572d.slice/crio-22c24f9dfc759eb877b12037ab60a5d2470a88622a385a92b3e433384428f553 WatchSource:0}: Error finding container 22c24f9dfc759eb877b12037ab60a5d2470a88622a385a92b3e433384428f553: Status 404 returned error can't find the container with id 22c24f9dfc759eb877b12037ab60a5d2470a88622a385a92b3e433384428f553 Sep 17 01:57:51.081392 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:51.081299 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" event={"ID":"ab66b979-d535-4d31-bc5d-d0693cdc572d","Type":"ContainerStarted","Data":"22c24f9dfc759eb877b12037ab60a5d2470a88622a385a92b3e433384428f553"} Sep 17 01:57:52.089221 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:52.089181 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" event={"ID":"ab66b979-d535-4d31-bc5d-d0693cdc572d","Type":"ContainerStarted","Data":"b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7"} Sep 17 01:57:52.089637 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:52.089353 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:57:52.106857 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:52.106813 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" podStartSLOduration=1.809485658 podStartE2EDuration="2.106798647s" totalImagesPullingTime="297.312989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:57:52.104982377 +0000 UTC m=+1235.861749650" watchObservedRunningTime="2026-09-17 01:57:52.106798647 +0000 UTC m=+1235.863565891" Sep 17 01:57:58.102977 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:57:58.102940 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:58:28.689790 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:28.689753 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz"] Sep 17 01:58:28.690369 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:28.690038 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" podUID="ab66b979-d535-4d31-bc5d-d0693cdc572d" containerName="maas-api" containerID="cri-o://b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7" gracePeriod=30 Sep 17 01:58:28.963054 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:28.963027 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:58:28.990171 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:28.990140 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-metrics-certs\") pod \"ab66b979-d535-4d31-bc5d-d0693cdc572d\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " Sep 17 01:58:28.990303 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:28.990240 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab66b979-d535-4d31-bc5d-d0693cdc572d-openshift-service-ca\") pod \"ab66b979-d535-4d31-bc5d-d0693cdc572d\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " Sep 17 01:58:28.990303 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:28.990280 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h44b\" (UniqueName: \"kubernetes.io/projected/ab66b979-d535-4d31-bc5d-d0693cdc572d-kube-api-access-6h44b\") pod \"ab66b979-d535-4d31-bc5d-d0693cdc572d\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " Sep 17 01:58:28.991543 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:28.991481 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab66b979-d535-4d31-bc5d-d0693cdc572d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab66b979-d535-4d31-bc5d-d0693cdc572d" (UID: "ab66b979-d535-4d31-bc5d-d0693cdc572d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:58:28.994205 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:28.994149 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ab66b979-d535-4d31-bc5d-d0693cdc572d" (UID: "ab66b979-d535-4d31-bc5d-d0693cdc572d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:58:28.994205 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:28.994153 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab66b979-d535-4d31-bc5d-d0693cdc572d-kube-api-access-6h44b" (OuterVolumeSpecName: "kube-api-access-6h44b") pod "ab66b979-d535-4d31-bc5d-d0693cdc572d" (UID: "ab66b979-d535-4d31-bc5d-d0693cdc572d"). InnerVolumeSpecName "kube-api-access-6h44b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:58:29.091113 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.091084 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-maas-api-tls\") pod \"ab66b979-d535-4d31-bc5d-d0693cdc572d\" (UID: \"ab66b979-d535-4d31-bc5d-d0693cdc572d\") " Sep 17 01:58:29.091339 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.091327 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:58:29.091391 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.091341 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab66b979-d535-4d31-bc5d-d0693cdc572d-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:58:29.091391 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.091351 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h44b\" (UniqueName: \"kubernetes.io/projected/ab66b979-d535-4d31-bc5d-d0693cdc572d-kube-api-access-6h44b\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:58:29.093348 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.093319 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab66b979-d535-4d31-bc5d-d0693cdc572d" (UID: "ab66b979-d535-4d31-bc5d-d0693cdc572d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:58:29.192614 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.192554 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab66b979-d535-4d31-bc5d-d0693cdc572d-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 01:58:29.332261 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.332176 2586 generic.go:356] "Generic (PLEG): container finished" podID="ab66b979-d535-4d31-bc5d-d0693cdc572d" containerID="b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7" exitCode=0 Sep 17 01:58:29.332261 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.332256 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" Sep 17 01:58:29.332474 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.332272 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" event={"ID":"ab66b979-d535-4d31-bc5d-d0693cdc572d","Type":"ContainerDied","Data":"b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7"} Sep 17 01:58:29.332474 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.332334 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz" event={"ID":"ab66b979-d535-4d31-bc5d-d0693cdc572d","Type":"ContainerDied","Data":"22c24f9dfc759eb877b12037ab60a5d2470a88622a385a92b3e433384428f553"} Sep 17 01:58:29.332474 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.332361 2586 scope.go:117] "RemoveContainer" containerID="b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7" Sep 17 01:58:29.343372 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.343353 2586 scope.go:117] "RemoveContainer" containerID="b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7" Sep 17 01:58:29.343646 ip-10-0-143-174 kubenswrapper[2586]: E0917 01:58:29.343626 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7\": container with ID starting with b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7 not found: ID does not exist" containerID="b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7" Sep 17 01:58:29.343726 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.343657 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7"} err="failed to get container status \"b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7\": rpc error: code = NotFound desc = could not find container \"b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7\": container with ID starting with b466f9fa78784b2fc21d8b3a8d03ed04c26b9b2a95993f2a52dfabc65089f4f7 not found: ID does not exist" Sep 17 01:58:29.355201 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.355178 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz"] Sep 17 01:58:29.359170 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:29.359146 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-56574e4f-8694488759-mzsjz"] Sep 17 01:58:30.769079 ip-10-0-143-174 kubenswrapper[2586]: I0917 01:58:30.769045 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab66b979-d535-4d31-bc5d-d0693cdc572d" path="/var/lib/kubelet/pods/ab66b979-d535-4d31-bc5d-d0693cdc572d/volumes" Sep 17 02:00:00.141731 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.141653 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4"] Sep 17 02:00:00.142178 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.142162 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab66b979-d535-4d31-bc5d-d0693cdc572d" containerName="maas-api" Sep 17 02:00:00.142222 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.142180 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab66b979-d535-4d31-bc5d-d0693cdc572d" containerName="maas-api" Sep 17 02:00:00.142283 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.142274 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab66b979-d535-4d31-bc5d-d0693cdc572d" containerName="maas-api" Sep 17 02:00:00.145469 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.145454 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f"] Sep 17 02:00:00.145615 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.145572 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4" Sep 17 02:00:00.148090 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.148066 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:00:00.149326 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.149313 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f" Sep 17 02:00:00.160109 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.160083 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4" Sep 17 02:00:00.160320 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.160294 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4"] Sep 17 02:00:00.164548 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.164527 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f"] Sep 17 02:00:00.170274 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.170257 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f" Sep 17 02:00:00.452079 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.452035 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4"] Sep 17 02:00:00.454140 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:00:00.454112 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037de1ed_7ba2_44f7_aa19_d0af1e24f107.slice/crio-668c9d5530bf366732a5a1e224b47759daa7474300ac5c945ee82b006115a7f0 WatchSource:0}: Error finding container 668c9d5530bf366732a5a1e224b47759daa7474300ac5c945ee82b006115a7f0: Status 404 returned error can't find the container with id 668c9d5530bf366732a5a1e224b47759daa7474300ac5c945ee82b006115a7f0 Sep 17 02:00:00.454980 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:00:00.454958 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafe2171_1738_4cda_9246_1957326379d5.slice/crio-be0d4162da736a725e68771759f9020d0a35699c508aeb07eafb3ed2986a464d WatchSource:0}: Error finding container be0d4162da736a725e68771759f9020d0a35699c508aeb07eafb3ed2986a464d: Status 404 returned error can't find the container with id be0d4162da736a725e68771759f9020d0a35699c508aeb07eafb3ed2986a464d Sep 17 02:00:00.455617 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.455554 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f"] Sep 17 02:00:00.456333 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.456318 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 02:00:00.900027 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.899991 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4" event={"ID":"037de1ed-7ba2-44f7-aa19-d0af1e24f107","Type":"ContainerStarted","Data":"668c9d5530bf366732a5a1e224b47759daa7474300ac5c945ee82b006115a7f0"} Sep 17 02:00:00.901254 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:00.901229 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f" event={"ID":"eafe2171-1738-4cda-9246-1957326379d5","Type":"ContainerStarted","Data":"be0d4162da736a725e68771759f9020d0a35699c508aeb07eafb3ed2986a464d"} Sep 17 02:00:03.933318 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:03.933280 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4" event={"ID":"037de1ed-7ba2-44f7-aa19-d0af1e24f107","Type":"ContainerStarted","Data":"bb9a9b40986dc7b6a61cc3ab25ec42bee6a3f4809d4e0413b6c1c6e5024e51c3"} Sep 17 02:00:03.934947 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:03.934922 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f" event={"ID":"eafe2171-1738-4cda-9246-1957326379d5","Type":"ContainerStarted","Data":"5a0e7e4cd1c7bdf70ade7eae4e0d4e41d60977ed7aacf016b5b56e37a1d30b77"} Sep 17 02:00:03.949384 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:03.949339 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4" podStartSLOduration=1.2583251899999999 podStartE2EDuration="3.949326396s" totalImagesPullingTime="2.691001206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:00:03.94707181 +0000 UTC m=+1367.703839071" watchObservedRunningTime="2026-09-17 02:00:03.949326396 +0000 UTC m=+1367.706093639" Sep 17 02:00:03.962222 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:03.962185 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f" podStartSLOduration=1.267648464 podStartE2EDuration="3.962172928s" totalImagesPullingTime="2.694524464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:00:03.960834204 +0000 UTC m=+1367.717601459" watchObservedRunningTime="2026-09-17 02:00:03.962172928 +0000 UTC m=+1367.718940223" Sep 17 02:00:04.943938 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:04.943900 2586 generic.go:356] "Generic (PLEG): container finished" podID="eafe2171-1738-4cda-9246-1957326379d5" containerID="5a0e7e4cd1c7bdf70ade7eae4e0d4e41d60977ed7aacf016b5b56e37a1d30b77" exitCode=0 Sep 17 02:00:04.944384 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:04.943990 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f" event={"ID":"eafe2171-1738-4cda-9246-1957326379d5","Type":"ContainerDied","Data":"5a0e7e4cd1c7bdf70ade7eae4e0d4e41d60977ed7aacf016b5b56e37a1d30b77"} Sep 17 02:00:04.945628 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:04.945608 2586 generic.go:356] "Generic (PLEG): container finished" podID="037de1ed-7ba2-44f7-aa19-d0af1e24f107" containerID="bb9a9b40986dc7b6a61cc3ab25ec42bee6a3f4809d4e0413b6c1c6e5024e51c3" exitCode=0 Sep 17 02:00:04.945721 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:04.945685 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4" event={"ID":"037de1ed-7ba2-44f7-aa19-d0af1e24f107","Type":"ContainerDied","Data":"bb9a9b40986dc7b6a61cc3ab25ec42bee6a3f4809d4e0413b6c1c6e5024e51c3"} Sep 17 02:00:06.098680 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:06.098649 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4" Sep 17 02:00:06.135161 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:06.135131 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f" Sep 17 02:00:06.959150 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:06.959125 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4" Sep 17 02:00:06.959335 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:06.959157 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4" event={"ID":"037de1ed-7ba2-44f7-aa19-d0af1e24f107","Type":"ContainerDied","Data":"668c9d5530bf366732a5a1e224b47759daa7474300ac5c945ee82b006115a7f0"} Sep 17 02:00:06.959335 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:06.959218 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="668c9d5530bf366732a5a1e224b47759daa7474300ac5c945ee82b006115a7f0" Sep 17 02:00:06.960715 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:06.960695 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f" Sep 17 02:00:06.960834 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:06.960728 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f" event={"ID":"eafe2171-1738-4cda-9246-1957326379d5","Type":"ContainerDied","Data":"be0d4162da736a725e68771759f9020d0a35699c508aeb07eafb3ed2986a464d"} Sep 17 02:00:06.960834 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:06.960761 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be0d4162da736a725e68771759f9020d0a35699c508aeb07eafb3ed2986a464d" Sep 17 02:00:12.024189 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.024149 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c"] Sep 17 02:00:12.024884 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.024861 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafe2171-1738-4cda-9246-1957326379d5" containerName="cleanup" Sep 17 02:00:12.024972 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.024888 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafe2171-1738-4cda-9246-1957326379d5" containerName="cleanup" Sep 17 02:00:12.024972 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.024913 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037de1ed-7ba2-44f7-aa19-d0af1e24f107" containerName="cleanup" Sep 17 02:00:12.024972 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.024921 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="037de1ed-7ba2-44f7-aa19-d0af1e24f107" containerName="cleanup" Sep 17 02:00:12.025146 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.025063 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="037de1ed-7ba2-44f7-aa19-d0af1e24f107" containerName="cleanup" Sep 17 02:00:12.025146 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.025081 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafe2171-1738-4cda-9246-1957326379d5" containerName="cleanup" Sep 17 02:00:12.028681 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.028659 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.031325 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.031300 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4a1f5690-serving-cert\"" Sep 17 02:00:12.036909 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.036329 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c"] Sep 17 02:00:12.120842 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.120802 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-metrics-certs\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.121039 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.120856 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvfzr\" (UniqueName: \"kubernetes.io/projected/57c26d86-6d33-49ee-8ebe-f5314b27d948-kube-api-access-pvfzr\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.121039 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.120889 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c26d86-6d33-49ee-8ebe-f5314b27d948-openshift-service-ca\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.121039 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.120969 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-maas-api-tls\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.222352 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.222312 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-metrics-certs\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.222560 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.222391 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvfzr\" (UniqueName: \"kubernetes.io/projected/57c26d86-6d33-49ee-8ebe-f5314b27d948-kube-api-access-pvfzr\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.222560 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.222432 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c26d86-6d33-49ee-8ebe-f5314b27d948-openshift-service-ca\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.222560 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.222478 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-maas-api-tls\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.222834 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:00:12.222646 2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-serving-cert: secret "maas-api-e2e-mt-4a1f5690-serving-cert" not found Sep 17 02:00:12.222834 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:00:12.222717 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-maas-api-tls podName:57c26d86-6d33-49ee-8ebe-f5314b27d948 nodeName:}" failed. No retries permitted until 2026-09-17 02:00:12.722698805 +0000 UTC m=+1376.479466029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-maas-api-tls") pod "maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" (UID: "57c26d86-6d33-49ee-8ebe-f5314b27d948") : secret "maas-api-e2e-mt-4a1f5690-serving-cert" not found Sep 17 02:00:12.223274 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.223248 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c26d86-6d33-49ee-8ebe-f5314b27d948-openshift-service-ca\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.225102 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.225083 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-metrics-certs\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.232716 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.232692 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvfzr\" (UniqueName: \"kubernetes.io/projected/57c26d86-6d33-49ee-8ebe-f5314b27d948-kube-api-access-pvfzr\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.727943 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.727908 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-maas-api-tls\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.730474 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.730444 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-maas-api-tls\") pod \"maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:12.941510 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:12.941473 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:13.220373 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:13.220340 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c"] Sep 17 02:00:13.222130 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:00:13.222096 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c26d86_6d33_49ee_8ebe_f5314b27d948.slice/crio-8c7f5ec3662076a71690ed00a6e7f3acea22a31f4942be1784dbe8c64ab98ec5 WatchSource:0}: Error finding container 8c7f5ec3662076a71690ed00a6e7f3acea22a31f4942be1784dbe8c64ab98ec5: Status 404 returned error can't find the container with id 8c7f5ec3662076a71690ed00a6e7f3acea22a31f4942be1784dbe8c64ab98ec5 Sep 17 02:00:13.406153 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:13.406103 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4"] Sep 17 02:00:13.409325 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:13.409303 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38758b-29826840-x62b4"] Sep 17 02:00:13.445735 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:13.445703 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh"] Sep 17 02:00:13.445951 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:13.445925 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" podUID="2a144920-e52a-4694-ab4f-bc5a612917db" containerName="payload-pre-processing" containerID="cri-o://cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60" gracePeriod=30 Sep 17 02:00:13.747866 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:13.747731 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 02:00:13.942852 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:13.942771 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6fg2\" (UniqueName: \"kubernetes.io/projected/2a144920-e52a-4694-ab4f-bc5a612917db-kube-api-access-r6fg2\") pod \"2a144920-e52a-4694-ab4f-bc5a612917db\" (UID: \"2a144920-e52a-4694-ab4f-bc5a612917db\") " Sep 17 02:00:13.942852 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:13.942814 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a144920-e52a-4694-ab4f-bc5a612917db-plugins-config-volume\") pod \"2a144920-e52a-4694-ab4f-bc5a612917db\" (UID: \"2a144920-e52a-4694-ab4f-bc5a612917db\") " Sep 17 02:00:13.943289 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:13.943244 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a144920-e52a-4694-ab4f-bc5a612917db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a144920-e52a-4694-ab4f-bc5a612917db" (UID: "2a144920-e52a-4694-ab4f-bc5a612917db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:00:13.945155 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:13.945130 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a144920-e52a-4694-ab4f-bc5a612917db-kube-api-access-r6fg2" (OuterVolumeSpecName: "kube-api-access-r6fg2") pod "2a144920-e52a-4694-ab4f-bc5a612917db" (UID: "2a144920-e52a-4694-ab4f-bc5a612917db"). InnerVolumeSpecName "kube-api-access-r6fg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:14.008414 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.008382 2586 generic.go:356] "Generic (PLEG): container finished" podID="2a144920-e52a-4694-ab4f-bc5a612917db" containerID="cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60" exitCode=0 Sep 17 02:00:14.008559 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.008424 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" Sep 17 02:00:14.008559 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.008462 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" event={"ID":"2a144920-e52a-4694-ab4f-bc5a612917db","Type":"ContainerDied","Data":"cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60"} Sep 17 02:00:14.008559 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.008520 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh" event={"ID":"2a144920-e52a-4694-ab4f-bc5a612917db","Type":"ContainerDied","Data":"bef481beb7f3421b1e9311ce7001cba06f98c2725f9f0756255e0ef41fe919f5"} Sep 17 02:00:14.008559 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.008548 2586 scope.go:117] "RemoveContainer" containerID="cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60" Sep 17 02:00:14.010433 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.010407 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" event={"ID":"57c26d86-6d33-49ee-8ebe-f5314b27d948","Type":"ContainerStarted","Data":"23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe"} Sep 17 02:00:14.010554 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.010449 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" event={"ID":"57c26d86-6d33-49ee-8ebe-f5314b27d948","Type":"ContainerStarted","Data":"8c7f5ec3662076a71690ed00a6e7f3acea22a31f4942be1784dbe8c64ab98ec5"} Sep 17 02:00:14.010554 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.010474 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:14.018559 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.018542 2586 scope.go:117] "RemoveContainer" containerID="cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60" Sep 17 02:00:14.018892 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:00:14.018874 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60\": container with ID starting with cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60 not found: ID does not exist" containerID="cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60" Sep 17 02:00:14.018970 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.018899 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60"} err="failed to get container status \"cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60\": rpc error: code = NotFound desc = could not find container \"cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60\": container with ID starting with cc24fc828f22caba7b9617cc052fe911757d338391108dbbee50dc4cf424dc60 not found: ID does not exist" Sep 17 02:00:14.029551 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.029510 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" podStartSLOduration=1.6439497840000001 podStartE2EDuration="2.029497318s" totalImagesPullingTime="385.547534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:00:14.028790621 +0000 UTC m=+1377.785557872" watchObservedRunningTime="2026-09-17 02:00:14.029497318 +0000 UTC m=+1377.786264595" Sep 17 02:00:14.044478 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.044453 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6fg2\" (UniqueName: \"kubernetes.io/projected/2a144920-e52a-4694-ab4f-bc5a612917db-kube-api-access-r6fg2\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:14.044601 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.044482 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a144920-e52a-4694-ab4f-bc5a612917db-plugins-config-volume\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:14.054828 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.054797 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh"] Sep 17 02:00:14.054961 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.054849 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38758b-7d6c5b6b9d-f9bxh"] Sep 17 02:00:14.769031 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.768997 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="037de1ed-7ba2-44f7-aa19-d0af1e24f107" path="/var/lib/kubelet/pods/037de1ed-7ba2-44f7-aa19-d0af1e24f107/volumes" Sep 17 02:00:14.769389 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:14.769335 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a144920-e52a-4694-ab4f-bc5a612917db" path="/var/lib/kubelet/pods/2a144920-e52a-4694-ab4f-bc5a612917db/volumes" Sep 17 02:00:20.011175 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:20.011141 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd"] Sep 17 02:00:20.011688 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:20.011397 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" containerID="cri-o://fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6" gracePeriod=30 Sep 17 02:00:20.025399 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:20.025375 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:21.784133 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:21.784098 2586 patch_prober.go:28] interesting pod/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 02:00:21.784535 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:21.784159 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 02:00:25.269704 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.269657 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 02:00:25.348451 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348362 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-podinfo\") pod \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " Sep 17 02:00:25.348636 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348456 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istiod-ca-cert\") pod \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " Sep 17 02:00:25.348636 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348518 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-credential-socket\") pod \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " Sep 17 02:00:25.348636 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348542 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-data\") pod \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " Sep 17 02:00:25.348816 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348660 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-token\") pod \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " Sep 17 02:00:25.348816 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348706 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-certs\") pod \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " Sep 17 02:00:25.348816 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348745 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-socket\") pod \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " Sep 17 02:00:25.348816 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348785 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-envoy\") pod \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " Sep 17 02:00:25.348816 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348790 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" (UID: "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:25.349058 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348822 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtd46\" (UniqueName: \"kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-kube-api-access-jtd46\") pod \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\" (UID: \"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766\") " Sep 17 02:00:25.349058 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348901 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" (UID: "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:25.349058 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348923 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" (UID: "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:00:25.349058 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.348985 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" (UID: "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:25.349250 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.349221 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-credential-socket\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:25.349250 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.349239 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-data\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:25.349345 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.349252 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-socket\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:25.349345 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.349265 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istiod-ca-cert\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:25.349422 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.349354 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" (UID: "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:25.350830 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.350803 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" (UID: "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 02:00:25.351130 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.351104 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" (UID: "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:25.351463 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.351437 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" (UID: "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:25.351555 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.351528 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-kube-api-access-jtd46" (OuterVolumeSpecName: "kube-api-access-jtd46") pod "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" (UID: "0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766"). InnerVolumeSpecName "kube-api-access-jtd46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:25.450988 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.450950 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-token\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:25.450988 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.450987 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-workload-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:25.451263 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.451000 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-envoy\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:25.451263 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.451015 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtd46\" (UniqueName: \"kubernetes.io/projected/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-kube-api-access-jtd46\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:25.451263 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:25.451031 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766-istio-podinfo\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:26.092555 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.092515 2586 generic.go:356] "Generic (PLEG): container finished" podID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerID="fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6" exitCode=0 Sep 17 02:00:26.092749 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.092570 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" Sep 17 02:00:26.092749 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.092613 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" event={"ID":"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766","Type":"ContainerDied","Data":"fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6"} Sep 17 02:00:26.092749 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.092660 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd" event={"ID":"0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766","Type":"ContainerDied","Data":"ed4b20df9db4b1a6f9dafeb4aec3fd1a525178a7fa38884af50adb200e9aff23"} Sep 17 02:00:26.092749 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.092684 2586 scope.go:117] "RemoveContainer" containerID="fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6" Sep 17 02:00:26.103458 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.103437 2586 scope.go:117] "RemoveContainer" containerID="59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448" Sep 17 02:00:26.113012 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.112978 2586 scope.go:117] "RemoveContainer" containerID="fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6" Sep 17 02:00:26.113770 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:00:26.113738 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6\": container with ID starting with fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6 not found: ID does not exist" containerID="fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6" Sep 17 02:00:26.113886 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.113782 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6"} err="failed to get container status \"fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6\": rpc error: code = NotFound desc = could not find container \"fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6\": container with ID starting with fe4aceca49661454bace65f865d41944d721b86b6bae720da96991a43689a9a6 not found: ID does not exist" Sep 17 02:00:26.113886 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.113807 2586 scope.go:117] "RemoveContainer" containerID="59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448" Sep 17 02:00:26.114101 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:00:26.114075 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448\": container with ID starting with 59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448 not found: ID does not exist" containerID="59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448" Sep 17 02:00:26.114170 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.114109 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448"} err="failed to get container status \"59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448\": rpc error: code = NotFound desc = could not find container \"59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448\": container with ID starting with 59b4093d036d18d3ff0dd641a46905fed7af5e4df5c3e218bcb3da80e28d2448 not found: ID does not exist" Sep 17 02:00:26.116417 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.116392 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd"] Sep 17 02:00:26.122598 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.122554 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-38758b-openshift-default-6f6854846d-dk2dd"] Sep 17 02:00:26.769311 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:26.769278 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" path="/var/lib/kubelet/pods/0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766/volumes" Sep 17 02:00:31.075562 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.075520 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6"] Sep 17 02:00:31.076009 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.075779 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" podUID="f7482277-e9b3-4fd5-8d8d-73293a42c40c" containerName="maas-api" containerID="cri-o://842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef" gracePeriod=30 Sep 17 02:00:31.528774 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.528749 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 02:00:31.614978 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.614872 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-maas-api-tls\") pod \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " Sep 17 02:00:31.614978 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.614972 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-metrics-certs\") pod \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " Sep 17 02:00:31.615204 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.615076 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7482277-e9b3-4fd5-8d8d-73293a42c40c-openshift-service-ca\") pod \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " Sep 17 02:00:31.615204 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.615101 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l226p\" (UniqueName: \"kubernetes.io/projected/f7482277-e9b3-4fd5-8d8d-73293a42c40c-kube-api-access-l226p\") pod \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\" (UID: \"f7482277-e9b3-4fd5-8d8d-73293a42c40c\") " Sep 17 02:00:31.615443 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.615415 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7482277-e9b3-4fd5-8d8d-73293a42c40c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7482277-e9b3-4fd5-8d8d-73293a42c40c" (UID: "f7482277-e9b3-4fd5-8d8d-73293a42c40c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:00:31.617250 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.617221 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f7482277-e9b3-4fd5-8d8d-73293a42c40c" (UID: "f7482277-e9b3-4fd5-8d8d-73293a42c40c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:00:31.617350 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.617262 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7482277-e9b3-4fd5-8d8d-73293a42c40c" (UID: "f7482277-e9b3-4fd5-8d8d-73293a42c40c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:00:31.617350 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.617277 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7482277-e9b3-4fd5-8d8d-73293a42c40c-kube-api-access-l226p" (OuterVolumeSpecName: "kube-api-access-l226p") pod "f7482277-e9b3-4fd5-8d8d-73293a42c40c" (UID: "f7482277-e9b3-4fd5-8d8d-73293a42c40c"). InnerVolumeSpecName "kube-api-access-l226p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:31.716469 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.716419 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7482277-e9b3-4fd5-8d8d-73293a42c40c-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:31.716469 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.716457 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l226p\" (UniqueName: \"kubernetes.io/projected/f7482277-e9b3-4fd5-8d8d-73293a42c40c-kube-api-access-l226p\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:31.716469 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.716471 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:31.716806 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:31.716483 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7482277-e9b3-4fd5-8d8d-73293a42c40c-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:32.135794 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:32.135759 2586 generic.go:356] "Generic (PLEG): container finished" podID="f7482277-e9b3-4fd5-8d8d-73293a42c40c" containerID="842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef" exitCode=0 Sep 17 02:00:32.136149 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:32.135803 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" Sep 17 02:00:32.136149 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:32.135841 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" event={"ID":"f7482277-e9b3-4fd5-8d8d-73293a42c40c","Type":"ContainerDied","Data":"842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef"} Sep 17 02:00:32.136149 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:32.135892 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6" event={"ID":"f7482277-e9b3-4fd5-8d8d-73293a42c40c","Type":"ContainerDied","Data":"c7b31c87e74ee5be405af55b00fd7e1a25fe6d51134041aeb6790d451164cf77"} Sep 17 02:00:32.136149 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:32.135915 2586 scope.go:117] "RemoveContainer" containerID="842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef" Sep 17 02:00:32.148881 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:32.148862 2586 scope.go:117] "RemoveContainer" containerID="842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef" Sep 17 02:00:32.149186 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:00:32.149162 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef\": container with ID starting with 842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef not found: ID does not exist" containerID="842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef" Sep 17 02:00:32.149258 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:32.149194 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef"} err="failed to get container status \"842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef\": rpc error: code = NotFound desc = could not find container \"842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef\": container with ID starting with 842bc7c726263067fba692b31797953f2b5c3bc73aa51ed9bddf1191d4c5bfef not found: ID does not exist" Sep 17 02:00:32.160743 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:32.160716 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6"] Sep 17 02:00:32.164148 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:32.164128 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c648-59496859cc-8tqr6"] Sep 17 02:00:32.768835 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:32.768805 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7482277-e9b3-4fd5-8d8d-73293a42c40c" path="/var/lib/kubelet/pods/f7482277-e9b3-4fd5-8d8d-73293a42c40c/volumes" Sep 17 02:00:38.150567 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:38.150528 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr"] Sep 17 02:00:38.151101 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:38.150803 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" containerID="cri-o://dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a" gracePeriod=30 Sep 17 02:00:41.134704 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:41.134667 2586 patch_prober.go:28] interesting pod/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 02:00:41.135069 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:41.134734 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 02:00:43.397036 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.397010 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 02:00:43.539470 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539384 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-data\") pod \"5249a534-f346-46eb-9e0e-e8c275efdf27\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " Sep 17 02:00:43.539470 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539415 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5249a534-f346-46eb-9e0e-e8c275efdf27-istiod-ca-cert\") pod \"5249a534-f346-46eb-9e0e-e8c275efdf27\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " Sep 17 02:00:43.539470 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539438 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-podinfo\") pod \"5249a534-f346-46eb-9e0e-e8c275efdf27\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " Sep 17 02:00:43.539790 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539511 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-token\") pod \"5249a534-f346-46eb-9e0e-e8c275efdf27\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " Sep 17 02:00:43.539790 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539551 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-envoy\") pod \"5249a534-f346-46eb-9e0e-e8c275efdf27\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " Sep 17 02:00:43.539790 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539601 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-credential-socket\") pod \"5249a534-f346-46eb-9e0e-e8c275efdf27\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " Sep 17 02:00:43.539790 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539630 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj6hj\" (UniqueName: \"kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-kube-api-access-jj6hj\") pod \"5249a534-f346-46eb-9e0e-e8c275efdf27\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " Sep 17 02:00:43.539790 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539661 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-socket\") pod \"5249a534-f346-46eb-9e0e-e8c275efdf27\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " Sep 17 02:00:43.539790 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539678 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-certs\") pod \"5249a534-f346-46eb-9e0e-e8c275efdf27\" (UID: \"5249a534-f346-46eb-9e0e-e8c275efdf27\") " Sep 17 02:00:43.539790 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539737 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-data" (OuterVolumeSpecName: "istio-data") pod "5249a534-f346-46eb-9e0e-e8c275efdf27" (UID: "5249a534-f346-46eb-9e0e-e8c275efdf27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:43.540125 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539852 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5249a534-f346-46eb-9e0e-e8c275efdf27-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5249a534-f346-46eb-9e0e-e8c275efdf27" (UID: "5249a534-f346-46eb-9e0e-e8c275efdf27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:00:43.540125 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.539989 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5249a534-f346-46eb-9e0e-e8c275efdf27" (UID: "5249a534-f346-46eb-9e0e-e8c275efdf27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:43.540125 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.540024 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5249a534-f346-46eb-9e0e-e8c275efdf27" (UID: "5249a534-f346-46eb-9e0e-e8c275efdf27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:43.540125 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.540059 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-data\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:43.540125 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.540079 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5249a534-f346-46eb-9e0e-e8c275efdf27-istiod-ca-cert\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:43.540370 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.540148 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5249a534-f346-46eb-9e0e-e8c275efdf27" (UID: "5249a534-f346-46eb-9e0e-e8c275efdf27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:43.541971 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.541933 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-token" (OuterVolumeSpecName: "istio-token") pod "5249a534-f346-46eb-9e0e-e8c275efdf27" (UID: "5249a534-f346-46eb-9e0e-e8c275efdf27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:43.541971 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.541953 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5249a534-f346-46eb-9e0e-e8c275efdf27" (UID: "5249a534-f346-46eb-9e0e-e8c275efdf27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:43.542112 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.542037 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5249a534-f346-46eb-9e0e-e8c275efdf27" (UID: "5249a534-f346-46eb-9e0e-e8c275efdf27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 02:00:43.542295 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.542270 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-kube-api-access-jj6hj" (OuterVolumeSpecName: "kube-api-access-jj6hj") pod "5249a534-f346-46eb-9e0e-e8c275efdf27" (UID: "5249a534-f346-46eb-9e0e-e8c275efdf27"). InnerVolumeSpecName "kube-api-access-jj6hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:43.640749 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.640713 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-podinfo\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:43.640749 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.640742 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-token\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:43.640749 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.640751 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-istio-envoy\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:43.640749 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.640760 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-credential-socket\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:43.641009 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.640769 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj6hj\" (UniqueName: \"kubernetes.io/projected/5249a534-f346-46eb-9e0e-e8c275efdf27-kube-api-access-jj6hj\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:43.641009 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.640777 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-socket\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:43.641009 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:43.640786 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5249a534-f346-46eb-9e0e-e8c275efdf27-workload-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:44.218632 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.218555 2586 generic.go:356] "Generic (PLEG): container finished" podID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerID="dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a" exitCode=0 Sep 17 02:00:44.218632 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.218611 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" event={"ID":"5249a534-f346-46eb-9e0e-e8c275efdf27","Type":"ContainerDied","Data":"dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a"} Sep 17 02:00:44.218632 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.218638 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" Sep 17 02:00:44.218965 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.218673 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr" event={"ID":"5249a534-f346-46eb-9e0e-e8c275efdf27","Type":"ContainerDied","Data":"607e2731c0f1fcf76c551668a4ee1c5ed429843518111af9ba05be74fa0c0c2a"} Sep 17 02:00:44.218965 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.218698 2586 scope.go:117] "RemoveContainer" containerID="dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a" Sep 17 02:00:44.229812 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.229790 2586 scope.go:117] "RemoveContainer" containerID="105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3" Sep 17 02:00:44.247091 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.241820 2586 scope.go:117] "RemoveContainer" containerID="dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a" Sep 17 02:00:44.247091 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:00:44.243477 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a\": container with ID starting with dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a not found: ID does not exist" containerID="dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a" Sep 17 02:00:44.247091 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.243519 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a"} err="failed to get container status \"dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a\": rpc error: code = NotFound desc = could not find container \"dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a\": container with ID starting with dd1b247e0c266525872df18c98ee4e470f62f56c2e80f5397a8799497438bb3a not found: ID does not exist" Sep 17 02:00:44.247091 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.243549 2586 scope.go:117] "RemoveContainer" containerID="105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3" Sep 17 02:00:44.247091 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:00:44.243910 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3\": container with ID starting with 105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3 not found: ID does not exist" containerID="105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3" Sep 17 02:00:44.247091 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.243943 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3"} err="failed to get container status \"105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3\": rpc error: code = NotFound desc = could not find container \"105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3\": container with ID starting with 105b84cf6ca6f94e1dc2dbcfa6cb7668ab6a5fcdb93e35ba800d385b95437fb3 not found: ID does not exist" Sep 17 02:00:44.247430 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.247082 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr"] Sep 17 02:00:44.250692 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.250668 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-55c648-openshift-default-74b87f48c8-f6plr"] Sep 17 02:00:44.768081 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:44.768048 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" path="/var/lib/kubelet/pods/5249a534-f346-46eb-9e0e-e8c275efdf27/volumes" Sep 17 02:00:45.022354 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.022264 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5"] Sep 17 02:00:45.022979 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.022954 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" Sep 17 02:00:45.022979 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.022981 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" Sep 17 02:00:45.023183 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023002 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" Sep 17 02:00:45.023183 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023025 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" Sep 17 02:00:45.023183 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023056 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a144920-e52a-4694-ab4f-bc5a612917db" containerName="payload-pre-processing" Sep 17 02:00:45.023183 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023065 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a144920-e52a-4694-ab4f-bc5a612917db" containerName="payload-pre-processing" Sep 17 02:00:45.023183 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023084 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7482277-e9b3-4fd5-8d8d-73293a42c40c" containerName="maas-api" Sep 17 02:00:45.023183 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023092 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7482277-e9b3-4fd5-8d8d-73293a42c40c" containerName="maas-api" Sep 17 02:00:45.023183 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023104 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" Sep 17 02:00:45.023183 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023114 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" Sep 17 02:00:45.023183 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023132 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" Sep 17 02:00:45.023183 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023140 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" Sep 17 02:00:45.023747 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023234 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" Sep 17 02:00:45.023747 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023245 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6d4a6f-3f4c-4f52-a7c6-5dfa802d7766" containerName="istio-proxy" Sep 17 02:00:45.023747 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023255 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7482277-e9b3-4fd5-8d8d-73293a42c40c" containerName="maas-api" Sep 17 02:00:45.023747 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023266 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" Sep 17 02:00:45.023747 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023274 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a144920-e52a-4694-ab4f-bc5a612917db" containerName="payload-pre-processing" Sep 17 02:00:45.023747 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.023285 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5249a534-f346-46eb-9e0e-e8c275efdf27" containerName="istio-proxy" Sep 17 02:00:45.027786 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.027765 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.030325 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.030094 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-67a23a-serving-cert\"" Sep 17 02:00:45.035273 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.035237 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5"] Sep 17 02:00:45.157565 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.157529 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clkmm\" (UniqueName: \"kubernetes.io/projected/d3094a48-93ce-408f-8b9a-9b30e537c565-kube-api-access-clkmm\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.157813 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.157702 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.157813 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.157741 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3094a48-93ce-408f-8b9a-9b30e537c565-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.157813 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.157763 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.258977 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.258943 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3094a48-93ce-408f-8b9a-9b30e537c565-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.258977 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.258978 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.259224 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.259058 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clkmm\" (UniqueName: \"kubernetes.io/projected/d3094a48-93ce-408f-8b9a-9b30e537c565-kube-api-access-clkmm\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.259224 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.259170 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.259709 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.259687 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3094a48-93ce-408f-8b9a-9b30e537c565-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.261612 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.261595 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.261715 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.261683 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.266299 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.266278 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clkmm\" (UniqueName: \"kubernetes.io/projected/d3094a48-93ce-408f-8b9a-9b30e537c565-kube-api-access-clkmm\") pod \"maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.344399 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.344305 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:45.626774 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:45.626748 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5"] Sep 17 02:00:45.627914 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:00:45.627886 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3094a48_93ce_408f_8b9a_9b30e537c565.slice/crio-7cb0fd3c135287155ab7f42b3d12f1aa620c41508fad27d2260b96e750b95302 WatchSource:0}: Error finding container 7cb0fd3c135287155ab7f42b3d12f1aa620c41508fad27d2260b96e750b95302: Status 404 returned error can't find the container with id 7cb0fd3c135287155ab7f42b3d12f1aa620c41508fad27d2260b96e750b95302 Sep 17 02:00:46.238219 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:46.238183 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" event={"ID":"d3094a48-93ce-408f-8b9a-9b30e537c565","Type":"ContainerStarted","Data":"653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65"} Sep 17 02:00:46.238634 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:46.238234 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" event={"ID":"d3094a48-93ce-408f-8b9a-9b30e537c565","Type":"ContainerStarted","Data":"7cb0fd3c135287155ab7f42b3d12f1aa620c41508fad27d2260b96e750b95302"} Sep 17 02:00:46.238634 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:46.238324 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:00:46.256451 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:46.256323 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" podStartSLOduration=0.880015564 podStartE2EDuration="1.256299977s" totalImagesPullingTime="376.284413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:00:46.255221574 +0000 UTC m=+1410.011988843" watchObservedRunningTime="2026-09-17 02:00:46.256299977 +0000 UTC m=+1410.013067223" Sep 17 02:00:49.600104 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.600069 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c"] Sep 17 02:00:49.600595 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.600321 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" podUID="57c26d86-6d33-49ee-8ebe-f5314b27d948" containerName="maas-api" containerID="cri-o://23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe" gracePeriod=30 Sep 17 02:00:49.854657 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.854634 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:49.910840 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.910806 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c26d86-6d33-49ee-8ebe-f5314b27d948-openshift-service-ca\") pod \"57c26d86-6d33-49ee-8ebe-f5314b27d948\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " Sep 17 02:00:49.910840 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.910845 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-metrics-certs\") pod \"57c26d86-6d33-49ee-8ebe-f5314b27d948\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " Sep 17 02:00:49.911085 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.910879 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-maas-api-tls\") pod \"57c26d86-6d33-49ee-8ebe-f5314b27d948\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " Sep 17 02:00:49.911085 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.910984 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvfzr\" (UniqueName: \"kubernetes.io/projected/57c26d86-6d33-49ee-8ebe-f5314b27d948-kube-api-access-pvfzr\") pod \"57c26d86-6d33-49ee-8ebe-f5314b27d948\" (UID: \"57c26d86-6d33-49ee-8ebe-f5314b27d948\") " Sep 17 02:00:49.911199 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.911143 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c26d86-6d33-49ee-8ebe-f5314b27d948-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57c26d86-6d33-49ee-8ebe-f5314b27d948" (UID: "57c26d86-6d33-49ee-8ebe-f5314b27d948"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:00:49.911362 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.911338 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c26d86-6d33-49ee-8ebe-f5314b27d948-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:49.913238 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.913185 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57c26d86-6d33-49ee-8ebe-f5314b27d948" (UID: "57c26d86-6d33-49ee-8ebe-f5314b27d948"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:00:49.913344 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.913275 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c26d86-6d33-49ee-8ebe-f5314b27d948-kube-api-access-pvfzr" (OuterVolumeSpecName: "kube-api-access-pvfzr") pod "57c26d86-6d33-49ee-8ebe-f5314b27d948" (UID: "57c26d86-6d33-49ee-8ebe-f5314b27d948"). InnerVolumeSpecName "kube-api-access-pvfzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:49.913344 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:49.913273 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57c26d86-6d33-49ee-8ebe-f5314b27d948" (UID: "57c26d86-6d33-49ee-8ebe-f5314b27d948"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:00:50.012289 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.012256 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:50.012289 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.012284 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvfzr\" (UniqueName: \"kubernetes.io/projected/57c26d86-6d33-49ee-8ebe-f5314b27d948-kube-api-access-pvfzr\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:50.012289 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.012293 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c26d86-6d33-49ee-8ebe-f5314b27d948-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:00:50.273866 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.273782 2586 generic.go:356] "Generic (PLEG): container finished" podID="57c26d86-6d33-49ee-8ebe-f5314b27d948" containerID="23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe" exitCode=0 Sep 17 02:00:50.273866 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.273820 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" event={"ID":"57c26d86-6d33-49ee-8ebe-f5314b27d948","Type":"ContainerDied","Data":"23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe"} Sep 17 02:00:50.273866 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.273839 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" Sep 17 02:00:50.274098 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.273884 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c" event={"ID":"57c26d86-6d33-49ee-8ebe-f5314b27d948","Type":"ContainerDied","Data":"8c7f5ec3662076a71690ed00a6e7f3acea22a31f4942be1784dbe8c64ab98ec5"} Sep 17 02:00:50.274098 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.273905 2586 scope.go:117] "RemoveContainer" containerID="23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe" Sep 17 02:00:50.284329 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.284311 2586 scope.go:117] "RemoveContainer" containerID="23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe" Sep 17 02:00:50.284615 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:00:50.284572 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe\": container with ID starting with 23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe not found: ID does not exist" containerID="23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe" Sep 17 02:00:50.284664 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.284618 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe"} err="failed to get container status \"23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe\": rpc error: code = NotFound desc = could not find container \"23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe\": container with ID starting with 23934ac17e9e6e9c78c8fef744d9e3576285054e1bf895c280623f7d332a8bfe not found: ID does not exist" Sep 17 02:00:50.298520 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.298496 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c"] Sep 17 02:00:50.302233 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.302212 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4a1f5690-77b9758f65-sgc8c"] Sep 17 02:00:50.768439 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:50.768407 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57c26d86-6d33-49ee-8ebe-f5314b27d948" path="/var/lib/kubelet/pods/57c26d86-6d33-49ee-8ebe-f5314b27d948/volumes" Sep 17 02:00:52.250006 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:00:52.249967 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:01:14.527377 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.527327 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt"] Sep 17 02:01:14.528215 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.528195 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c26d86-6d33-49ee-8ebe-f5314b27d948" containerName="maas-api" Sep 17 02:01:14.528269 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.528221 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c26d86-6d33-49ee-8ebe-f5314b27d948" containerName="maas-api" Sep 17 02:01:14.528366 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.528353 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c26d86-6d33-49ee-8ebe-f5314b27d948" containerName="maas-api" Sep 17 02:01:14.547003 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.546968 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt"] Sep 17 02:01:14.547179 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.547085 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.549704 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.549674 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f03337-serving-cert\"" Sep 17 02:01:14.647445 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.647414 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6c603a-c0a4-4167-88a2-edcf5de51689-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.647445 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.647449 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5zh\" (UniqueName: \"kubernetes.io/projected/de6c603a-c0a4-4167-88a2-edcf5de51689-kube-api-access-8k5zh\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.647730 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.647475 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.647730 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.647495 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.749105 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.749063 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6c603a-c0a4-4167-88a2-edcf5de51689-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.749105 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.749100 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5zh\" (UniqueName: \"kubernetes.io/projected/de6c603a-c0a4-4167-88a2-edcf5de51689-kube-api-access-8k5zh\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.749329 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.749123 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.749329 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.749149 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.749329 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:01:14.749307 2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-serving-cert: secret "maas-api-e2e-shared-b-w2-f03337-serving-cert" not found Sep 17 02:01:14.749429 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:01:14.749390 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-maas-api-tls podName:de6c603a-c0a4-4167-88a2-edcf5de51689 nodeName:}" failed. No retries permitted until 2026-09-17 02:01:15.249369507 +0000 UTC m=+1439.006136733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-maas-api-tls") pod "maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" (UID: "de6c603a-c0a4-4167-88a2-edcf5de51689") : secret "maas-api-e2e-shared-b-w2-f03337-serving-cert" not found Sep 17 02:01:14.749830 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.749811 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6c603a-c0a4-4167-88a2-edcf5de51689-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.751729 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.751713 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:14.759297 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:14.759273 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5zh\" (UniqueName: \"kubernetes.io/projected/de6c603a-c0a4-4167-88a2-edcf5de51689-kube-api-access-8k5zh\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:15.255298 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:15.255261 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:15.258459 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:15.258428 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:15.460575 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:15.460536 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:15.733420 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:15.733390 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt"] Sep 17 02:01:15.734056 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:01:15.734030 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6c603a_c0a4_4167_88a2_edcf5de51689.slice/crio-14059e9a0635c72fd97227de1ad08df585aeaf61de5c27345fa9f4c034b4ecee WatchSource:0}: Error finding container 14059e9a0635c72fd97227de1ad08df585aeaf61de5c27345fa9f4c034b4ecee: Status 404 returned error can't find the container with id 14059e9a0635c72fd97227de1ad08df585aeaf61de5c27345fa9f4c034b4ecee Sep 17 02:01:16.440826 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:16.440789 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" event={"ID":"de6c603a-c0a4-4167-88a2-edcf5de51689","Type":"ContainerStarted","Data":"516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48"} Sep 17 02:01:16.440996 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:16.440843 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" event={"ID":"de6c603a-c0a4-4167-88a2-edcf5de51689","Type":"ContainerStarted","Data":"14059e9a0635c72fd97227de1ad08df585aeaf61de5c27345fa9f4c034b4ecee"} Sep 17 02:01:16.440996 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:16.440972 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:16.460077 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:16.460025 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" podStartSLOduration=2.099754559 podStartE2EDuration="2.460007266s" totalImagesPullingTime="360.252707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:16.459151636 +0000 UTC m=+1440.215918892" watchObservedRunningTime="2026-09-17 02:01:16.460007266 +0000 UTC m=+1440.216774509" Sep 17 02:01:19.415835 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.415732 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n"] Sep 17 02:01:19.430084 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.421737 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.430084 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.426037 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-c5da3b-serving-cert\"" Sep 17 02:01:19.430084 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.429774 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n"] Sep 17 02:01:19.603446 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.603405 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.603686 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.603458 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.603686 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.603618 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbw9k\" (UniqueName: \"kubernetes.io/projected/33e4b19c-e9e0-4d55-a915-8e2962dd822a-kube-api-access-bbw9k\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.603686 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.603683 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e4b19c-e9e0-4d55-a915-8e2962dd822a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.704612 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.704486 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbw9k\" (UniqueName: \"kubernetes.io/projected/33e4b19c-e9e0-4d55-a915-8e2962dd822a-kube-api-access-bbw9k\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.704612 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.704560 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e4b19c-e9e0-4d55-a915-8e2962dd822a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.704826 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.704676 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.704826 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.704713 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.704903 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:01:19.704848 2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-serving-cert: secret "maas-api-e2e-shared-a-w1-c5da3b-serving-cert" not found Sep 17 02:01:19.704945 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:01:19.704908 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-maas-api-tls podName:33e4b19c-e9e0-4d55-a915-8e2962dd822a nodeName:}" failed. No retries permitted until 2026-09-17 02:01:20.204889486 +0000 UTC m=+1443.961656709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-maas-api-tls") pod "maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" (UID: "33e4b19c-e9e0-4d55-a915-8e2962dd822a") : secret "maas-api-e2e-shared-a-w1-c5da3b-serving-cert" not found Sep 17 02:01:19.705260 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.705232 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e4b19c-e9e0-4d55-a915-8e2962dd822a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.707191 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.707169 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:19.713080 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:19.713056 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbw9k\" (UniqueName: \"kubernetes.io/projected/33e4b19c-e9e0-4d55-a915-8e2962dd822a-kube-api-access-bbw9k\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:20.210838 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:20.210801 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:20.213936 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:20.213909 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:20.336071 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:20.336027 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:20.611792 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:20.611765 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n"] Sep 17 02:01:20.613324 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:01:20.613300 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e4b19c_e9e0_4d55_a915_8e2962dd822a.slice/crio-40719b03705eae8da091180589e055b66df08acec635dab844a91bdce84a4268 WatchSource:0}: Error finding container 40719b03705eae8da091180589e055b66df08acec635dab844a91bdce84a4268: Status 404 returned error can't find the container with id 40719b03705eae8da091180589e055b66df08acec635dab844a91bdce84a4268 Sep 17 02:01:21.473396 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:21.473358 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" event={"ID":"33e4b19c-e9e0-4d55-a915-8e2962dd822a","Type":"ContainerStarted","Data":"c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636"} Sep 17 02:01:21.473563 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:21.473420 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" event={"ID":"33e4b19c-e9e0-4d55-a915-8e2962dd822a","Type":"ContainerStarted","Data":"40719b03705eae8da091180589e055b66df08acec635dab844a91bdce84a4268"} Sep 17 02:01:21.473563 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:21.473453 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:21.492726 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:21.492672 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" podStartSLOduration=2.041874744 podStartE2EDuration="2.49265292s" totalImagesPullingTime="450.778176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:21.490708384 +0000 UTC m=+1445.247475640" watchObservedRunningTime="2026-09-17 02:01:21.49265292 +0000 UTC m=+1445.249420164" Sep 17 02:01:22.453738 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:22.453707 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:01:27.486382 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:27.486352 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:01:52.825173 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:52.825137 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp"] Sep 17 02:01:52.829760 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:52.829724 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:52.831957 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:52.831933 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-a3912d-serving-cert\"" Sep 17 02:01:52.842037 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:52.842015 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp"] Sep 17 02:01:52.948948 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:52.948915 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:52.949113 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:52.948980 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbr8\" (UniqueName: \"kubernetes.io/projected/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-kube-api-access-7hbr8\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:52.949113 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:52.949033 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:52.949113 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:52.949100 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:53.050462 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:53.050423 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbr8\" (UniqueName: \"kubernetes.io/projected/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-kube-api-access-7hbr8\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:53.050666 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:53.050481 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:53.050666 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:53.050504 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:53.050666 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:53.050567 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:53.050844 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:01:53.050688 2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-serving-cert: secret "maas-api-e2e-shared-b-w1-a3912d-serving-cert" not found Sep 17 02:01:53.050844 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:01:53.050751 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-maas-api-tls podName:0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23 nodeName:}" failed. No retries permitted until 2026-09-17 02:01:53.550734089 +0000 UTC m=+1477.307501311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-maas-api-tls") pod "maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" (UID: "0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23") : secret "maas-api-e2e-shared-b-w1-a3912d-serving-cert" not found Sep 17 02:01:53.051259 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:53.051233 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:53.053119 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:53.053101 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:53.058846 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:53.058810 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbr8\" (UniqueName: \"kubernetes.io/projected/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-kube-api-access-7hbr8\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:53.554517 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:53.554485 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:53.557607 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:53.557568 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:53.743543 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:53.743509 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:54.023839 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:54.023800 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp"] Sep 17 02:01:54.686626 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:54.686516 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" event={"ID":"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23","Type":"ContainerStarted","Data":"018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e"} Sep 17 02:01:54.686626 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:54.686566 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" event={"ID":"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23","Type":"ContainerStarted","Data":"4495ecaf128e139196bdf16e6c7fea78c975cd10a57af97e38502971a976b8d7"} Sep 17 02:01:54.686626 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:54.686619 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:01:54.704364 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:54.704308 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" podStartSLOduration=2.323617506 podStartE2EDuration="2.704292072s" totalImagesPullingTime="380.674566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:54.702285004 +0000 UTC m=+1478.459052259" watchObservedRunningTime="2026-09-17 02:01:54.704292072 +0000 UTC m=+1478.461059315" Sep 17 02:01:56.563647 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.563617 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc"] Sep 17 02:01:56.567319 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.567302 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.571047 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.570852 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-67a23a\"/\"default-dockercfg-47wrb\"" Sep 17 02:01:56.571047 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.570866 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-67a23a\"/\"kube-root-ca.crt\"" Sep 17 02:01:56.571047 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.570967 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-67a23a\"/\"auth-test-model-67a23a-kserve-self-signed-certs\"" Sep 17 02:01:56.571307 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.571057 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-67a23a\"/\"openshift-service-ca.crt\"" Sep 17 02:01:56.575337 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.575311 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc"] Sep 17 02:01:56.582189 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.582157 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbb6564-5641-4114-8625-c6d058ae7ad7-tls-certs\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.582325 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.582268 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-tmp-dir\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.582325 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.582314 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-dshm\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.582461 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.582341 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-model-cache\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.582461 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.582373 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-home\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.582571 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.582501 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xrm\" (UniqueName: \"kubernetes.io/projected/adbb6564-5641-4114-8625-c6d058ae7ad7-kube-api-access-h6xrm\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.683629 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.683574 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbb6564-5641-4114-8625-c6d058ae7ad7-tls-certs\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.683796 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.683683 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-tmp-dir\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.683796 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.683719 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-dshm\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.683796 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.683742 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-model-cache\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.683796 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.683776 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-home\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.683972 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.683906 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xrm\" (UniqueName: \"kubernetes.io/projected/adbb6564-5641-4114-8625-c6d058ae7ad7-kube-api-access-h6xrm\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.684159 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.684135 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-tmp-dir\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.684278 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.684181 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-model-cache\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.684278 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.684197 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-home\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.686551 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.686527 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-dshm\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.686748 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.686731 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbb6564-5641-4114-8625-c6d058ae7ad7-tls-certs\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.691127 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.691110 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xrm\" (UniqueName: \"kubernetes.io/projected/adbb6564-5641-4114-8625-c6d058ae7ad7-kube-api-access-h6xrm\") pod \"auth-test-model-67a23a-kserve-6647f74f5f-wv9cc\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:56.878174 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:56.878138 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:01:57.159035 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:57.159005 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc"] Sep 17 02:01:57.159537 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:01:57.159512 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadbb6564_5641_4114_8625_c6d058ae7ad7.slice/crio-a594559415441184cf3394138f4bd71eb4dbbc0beb0971bb4cd2aa0bd5191655 WatchSource:0}: Error finding container a594559415441184cf3394138f4bd71eb4dbbc0beb0971bb4cd2aa0bd5191655: Status 404 returned error can't find the container with id a594559415441184cf3394138f4bd71eb4dbbc0beb0971bb4cd2aa0bd5191655 Sep 17 02:01:57.707874 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:57.707836 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" event={"ID":"adbb6564-5641-4114-8625-c6d058ae7ad7","Type":"ContainerStarted","Data":"f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2"} Sep 17 02:01:57.708240 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:57.707887 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" event={"ID":"adbb6564-5641-4114-8625-c6d058ae7ad7","Type":"ContainerStarted","Data":"a594559415441184cf3394138f4bd71eb4dbbc0beb0971bb4cd2aa0bd5191655"} Sep 17 02:01:57.724684 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:01:57.724630 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" podStartSLOduration=1.7246155600000002 podStartE2EDuration="1.72461556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:57.722483567 +0000 UTC m=+1481.479250811" watchObservedRunningTime="2026-09-17 02:01:57.72461556 +0000 UTC m=+1481.481382835" Sep 17 02:02:00.699286 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:00.699256 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:02:06.878664 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:06.878620 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:02:06.879146 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:06.878769 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:02:06.891540 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:06.891511 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:02:07.786550 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:07.786520 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:02:16.879494 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:16.879462 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 02:02:16.884344 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:16.884321 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 02:02:20.295366 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:20.295331 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10798-484fe6"] Sep 17 02:02:20.299105 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:20.299088 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-484fe6" Sep 17 02:02:20.301652 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:20.301626 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7c2dt\"" Sep 17 02:02:20.304916 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:20.304891 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10798-484fe6"] Sep 17 02:02:20.427071 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:20.427039 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5g77\" (UniqueName: \"kubernetes.io/projected/e992ebda-48b8-43a5-8dcd-d1254229f598-kube-api-access-x5g77\") pod \"test-curl-10798-484fe6\" (UID: \"e992ebda-48b8-43a5-8dcd-d1254229f598\") " pod="opendatahub/test-curl-10798-484fe6" Sep 17 02:02:20.528035 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:20.528004 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5g77\" (UniqueName: \"kubernetes.io/projected/e992ebda-48b8-43a5-8dcd-d1254229f598-kube-api-access-x5g77\") pod \"test-curl-10798-484fe6\" (UID: \"e992ebda-48b8-43a5-8dcd-d1254229f598\") " pod="opendatahub/test-curl-10798-484fe6" Sep 17 02:02:20.535881 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:20.535853 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5g77\" (UniqueName: \"kubernetes.io/projected/e992ebda-48b8-43a5-8dcd-d1254229f598-kube-api-access-x5g77\") pod \"test-curl-10798-484fe6\" (UID: \"e992ebda-48b8-43a5-8dcd-d1254229f598\") " pod="opendatahub/test-curl-10798-484fe6" Sep 17 02:02:20.609804 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:20.609781 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-484fe6" Sep 17 02:02:20.880741 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:20.880713 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10798-484fe6"] Sep 17 02:02:20.881684 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:02:20.881654 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode992ebda_48b8_43a5_8dcd_d1254229f598.slice/crio-8b6e4a73638835723eb9440eddbba89cd43316975f07240b7c87258af9a93792 WatchSource:0}: Error finding container 8b6e4a73638835723eb9440eddbba89cd43316975f07240b7c87258af9a93792: Status 404 returned error can't find the container with id 8b6e4a73638835723eb9440eddbba89cd43316975f07240b7c87258af9a93792 Sep 17 02:02:21.862912 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:21.862874 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10798-484fe6" event={"ID":"e992ebda-48b8-43a5-8dcd-d1254229f598","Type":"ContainerStarted","Data":"8b6e4a73638835723eb9440eddbba89cd43316975f07240b7c87258af9a93792"} Sep 17 02:02:25.889352 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:25.889319 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10798-484fe6" event={"ID":"e992ebda-48b8-43a5-8dcd-d1254229f598","Type":"ContainerStarted","Data":"377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50"} Sep 17 02:02:25.905989 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:25.905943 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10798-484fe6" podStartSLOduration=1.466217474 podStartE2EDuration="5.905924412s" totalImagesPullingTime="4.439706938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:02:25.903892836 +0000 UTC m=+1509.660660098" watchObservedRunningTime="2026-09-17 02:02:25.905924412 +0000 UTC m=+1509.662691657" Sep 17 02:02:26.513490 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.513443 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10798-484fe6"] Sep 17 02:02:26.523432 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.523398 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10798-484fe6"] Sep 17 02:02:26.649659 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.649627 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10798-47f79c"] Sep 17 02:02:26.650134 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.650120 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e992ebda-48b8-43a5-8dcd-d1254229f598" containerName="test-curl-10798-484fe6" Sep 17 02:02:26.650134 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.650134 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e992ebda-48b8-43a5-8dcd-d1254229f598" containerName="test-curl-10798-484fe6" Sep 17 02:02:26.650253 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.650222 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e992ebda-48b8-43a5-8dcd-d1254229f598" containerName="test-curl-10798-484fe6" Sep 17 02:02:26.653669 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.653654 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-47f79c" Sep 17 02:02:26.660104 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.660065 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10798-47f79c"] Sep 17 02:02:26.792842 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.792806 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg7c4\" (UniqueName: \"kubernetes.io/projected/6f278522-fb91-465b-b503-6761d7d1058d-kube-api-access-vg7c4\") pod \"test-curl-10798-47f79c\" (UID: \"6f278522-fb91-465b-b503-6761d7d1058d\") " pod="opendatahub/test-curl-10798-47f79c" Sep 17 02:02:26.894079 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.894045 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg7c4\" (UniqueName: \"kubernetes.io/projected/6f278522-fb91-465b-b503-6761d7d1058d-kube-api-access-vg7c4\") pod \"test-curl-10798-47f79c\" (UID: \"6f278522-fb91-465b-b503-6761d7d1058d\") " pod="opendatahub/test-curl-10798-47f79c" Sep 17 02:02:26.895614 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:02:26.895595 2586 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10798-484fe6" container="test-curl-10798-484fe6" Sep 17 02:02:26.902137 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.902079 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg7c4\" (UniqueName: \"kubernetes.io/projected/6f278522-fb91-465b-b503-6761d7d1058d-kube-api-access-vg7c4\") pod \"test-curl-10798-47f79c\" (UID: \"6f278522-fb91-465b-b503-6761d7d1058d\") " pod="opendatahub/test-curl-10798-47f79c" Sep 17 02:02:26.966016 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:26.965977 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-47f79c" Sep 17 02:02:27.242142 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:27.242114 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10798-47f79c"] Sep 17 02:02:27.243696 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:02:27.243663 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f278522_fb91_465b_b503_6761d7d1058d.slice/crio-4c83a385d93526309304f32f8c8d8a06115e722633624fb5ba357e1095033eaa WatchSource:0}: Error finding container 4c83a385d93526309304f32f8c8d8a06115e722633624fb5ba357e1095033eaa: Status 404 returned error can't find the container with id 4c83a385d93526309304f32f8c8d8a06115e722633624fb5ba357e1095033eaa Sep 17 02:02:27.903889 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:27.903855 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10798-47f79c" event={"ID":"6f278522-fb91-465b-b503-6761d7d1058d","Type":"ContainerStarted","Data":"964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3"} Sep 17 02:02:27.904298 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:27.903899 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10798-47f79c" event={"ID":"6f278522-fb91-465b-b503-6761d7d1058d","Type":"ContainerStarted","Data":"4c83a385d93526309304f32f8c8d8a06115e722633624fb5ba357e1095033eaa"} Sep 17 02:02:27.904298 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:27.904030 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10798-484fe6" podUID="e992ebda-48b8-43a5-8dcd-d1254229f598" containerName="test-curl-10798-484fe6" containerID="cri-o://377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50" gracePeriod=2 Sep 17 02:02:27.917229 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:27.917178 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10798-47f79c" podStartSLOduration=1.9171622579999998 podStartE2EDuration="1.917162258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:02:27.91672739 +0000 UTC m=+1511.673494638" watchObservedRunningTime="2026-09-17 02:02:27.917162258 +0000 UTC m=+1511.673929502" Sep 17 02:02:28.558159 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.558037 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10798-47f79c"] Sep 17 02:02:28.562550 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.562513 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10798-47f79c"] Sep 17 02:02:28.684942 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.684903 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10798-e3aece"] Sep 17 02:02:28.685427 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.685408 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f278522-fb91-465b-b503-6761d7d1058d" containerName="test-curl-10798-47f79c" Sep 17 02:02:28.685529 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.685430 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f278522-fb91-465b-b503-6761d7d1058d" containerName="test-curl-10798-47f79c" Sep 17 02:02:28.685633 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.685605 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f278522-fb91-465b-b503-6761d7d1058d" containerName="test-curl-10798-47f79c" Sep 17 02:02:28.689190 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.689167 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-e3aece" Sep 17 02:02:28.693804 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.693782 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10798-e3aece"] Sep 17 02:02:28.815970 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.815869 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9c4z\" (UniqueName: \"kubernetes.io/projected/20a54ba2-c7ca-45e1-b193-16a7d7047faa-kube-api-access-d9c4z\") pod \"test-curl-10798-e3aece\" (UID: \"20a54ba2-c7ca-45e1-b193-16a7d7047faa\") " pod="opendatahub/test-curl-10798-e3aece" Sep 17 02:02:28.909749 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:02:28.909721 2586 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10798-47f79c" container="test-curl-10798-47f79c" Sep 17 02:02:28.916951 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.916926 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c4z\" (UniqueName: \"kubernetes.io/projected/20a54ba2-c7ca-45e1-b193-16a7d7047faa-kube-api-access-d9c4z\") pod \"test-curl-10798-e3aece\" (UID: \"20a54ba2-c7ca-45e1-b193-16a7d7047faa\") " pod="opendatahub/test-curl-10798-e3aece" Sep 17 02:02:28.924620 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:28.924598 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9c4z\" (UniqueName: \"kubernetes.io/projected/20a54ba2-c7ca-45e1-b193-16a7d7047faa-kube-api-access-d9c4z\") pod \"test-curl-10798-e3aece\" (UID: \"20a54ba2-c7ca-45e1-b193-16a7d7047faa\") " pod="opendatahub/test-curl-10798-e3aece" Sep 17 02:02:29.000745 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:29.000708 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-e3aece" Sep 17 02:02:29.273389 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:29.273359 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10798-e3aece"] Sep 17 02:02:29.274350 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:02:29.274329 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20a54ba2_c7ca_45e1_b193_16a7d7047faa.slice/crio-75965bc6cccb3ef292b69e9f246820b98cdd197c9c1124804c9ce0dc6aa9167a WatchSource:0}: Error finding container 75965bc6cccb3ef292b69e9f246820b98cdd197c9c1124804c9ce0dc6aa9167a: Status 404 returned error can't find the container with id 75965bc6cccb3ef292b69e9f246820b98cdd197c9c1124804c9ce0dc6aa9167a Sep 17 02:02:29.917103 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:29.917071 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10798-e3aece" event={"ID":"20a54ba2-c7ca-45e1-b193-16a7d7047faa","Type":"ContainerStarted","Data":"7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048"} Sep 17 02:02:29.917443 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:29.917121 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10798-e3aece" event={"ID":"20a54ba2-c7ca-45e1-b193-16a7d7047faa","Type":"ContainerStarted","Data":"75965bc6cccb3ef292b69e9f246820b98cdd197c9c1124804c9ce0dc6aa9167a"} Sep 17 02:02:29.917443 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:29.917227 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10798-47f79c" podUID="6f278522-fb91-465b-b503-6761d7d1058d" containerName="test-curl-10798-47f79c" containerID="cri-o://964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3" gracePeriod=2 Sep 17 02:02:29.932206 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:29.932163 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10798-e3aece" podStartSLOduration=1.932147972 podStartE2EDuration="1.932147972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:02:29.930345669 +0000 UTC m=+1513.687112914" watchObservedRunningTime="2026-09-17 02:02:29.932147972 +0000 UTC m=+1513.688915216" Sep 17 02:02:30.085044 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.085025 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-484fe6" Sep 17 02:02:30.232115 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.232034 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5g77\" (UniqueName: \"kubernetes.io/projected/e992ebda-48b8-43a5-8dcd-d1254229f598-kube-api-access-x5g77\") pod \"e992ebda-48b8-43a5-8dcd-d1254229f598\" (UID: \"e992ebda-48b8-43a5-8dcd-d1254229f598\") " Sep 17 02:02:30.234230 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.234196 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e992ebda-48b8-43a5-8dcd-d1254229f598-kube-api-access-x5g77" (OuterVolumeSpecName: "kube-api-access-x5g77") pod "e992ebda-48b8-43a5-8dcd-d1254229f598" (UID: "e992ebda-48b8-43a5-8dcd-d1254229f598"). InnerVolumeSpecName "kube-api-access-x5g77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:02:30.333359 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.333336 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5g77\" (UniqueName: \"kubernetes.io/projected/e992ebda-48b8-43a5-8dcd-d1254229f598-kube-api-access-x5g77\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:02:30.550811 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.550731 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10798-e3aece"] Sep 17 02:02:30.555497 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.555467 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10798-e3aece"] Sep 17 02:02:30.673874 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.673826 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10798-ceb822"] Sep 17 02:02:30.674310 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.674298 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a54ba2-c7ca-45e1-b193-16a7d7047faa" containerName="test-curl-10798-e3aece" Sep 17 02:02:30.674367 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.674312 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a54ba2-c7ca-45e1-b193-16a7d7047faa" containerName="test-curl-10798-e3aece" Sep 17 02:02:30.674404 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.674397 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a54ba2-c7ca-45e1-b193-16a7d7047faa" containerName="test-curl-10798-e3aece" Sep 17 02:02:30.677670 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.677654 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-ceb822" Sep 17 02:02:30.683889 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.683863 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10798-ceb822"] Sep 17 02:02:30.769622 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.769572 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e992ebda-48b8-43a5-8dcd-d1254229f598" path="/var/lib/kubelet/pods/e992ebda-48b8-43a5-8dcd-d1254229f598/volumes" Sep 17 02:02:30.838664 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.838595 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9sg7\" (UniqueName: \"kubernetes.io/projected/6f507459-c744-4453-80f1-58e436c623d9-kube-api-access-q9sg7\") pod \"test-curl-10798-ceb822\" (UID: \"6f507459-c744-4453-80f1-58e436c623d9\") " pod="opendatahub/test-curl-10798-ceb822" Sep 17 02:02:30.925261 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.925225 2586 generic.go:356] "Generic (PLEG): container finished" podID="e992ebda-48b8-43a5-8dcd-d1254229f598" containerID="377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50" exitCode=137 Sep 17 02:02:30.925700 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.925267 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-484fe6" Sep 17 02:02:30.925700 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.925330 2586 scope.go:117] "RemoveContainer" containerID="377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50" Sep 17 02:02:30.925700 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:02:30.925427 2586 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10798-e3aece" container="test-curl-10798-e3aece" Sep 17 02:02:30.935362 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.935345 2586 scope.go:117] "RemoveContainer" containerID="377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50" Sep 17 02:02:30.935642 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:02:30.935618 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50\": container with ID starting with 377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50 not found: ID does not exist" containerID="377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50" Sep 17 02:02:30.935720 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.935653 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50"} err="failed to get container status \"377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50\": rpc error: code = NotFound desc = could not find container \"377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50\": container with ID starting with 377df2a1c760b4b141dfc46d462accbd73d34c8e2e350be18d70436da7a87f50 not found: ID does not exist" Sep 17 02:02:30.939249 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.939232 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9sg7\" (UniqueName: \"kubernetes.io/projected/6f507459-c744-4453-80f1-58e436c623d9-kube-api-access-q9sg7\") pod \"test-curl-10798-ceb822\" (UID: \"6f507459-c744-4453-80f1-58e436c623d9\") " pod="opendatahub/test-curl-10798-ceb822" Sep 17 02:02:30.946984 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.946962 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9sg7\" (UniqueName: \"kubernetes.io/projected/6f507459-c744-4453-80f1-58e436c623d9-kube-api-access-q9sg7\") pod \"test-curl-10798-ceb822\" (UID: \"6f507459-c744-4453-80f1-58e436c623d9\") " pod="opendatahub/test-curl-10798-ceb822" Sep 17 02:02:30.989610 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:30.989558 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-ceb822" Sep 17 02:02:31.265373 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:31.265341 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10798-ceb822"] Sep 17 02:02:31.267120 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:02:31.267094 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f507459_c744_4453_80f1_58e436c623d9.slice/crio-f2ab1ae998cc53236943895eac8d413105ed5d7e272ddfc806ad7cb75d9f4adf WatchSource:0}: Error finding container f2ab1ae998cc53236943895eac8d413105ed5d7e272ddfc806ad7cb75d9f4adf: Status 404 returned error can't find the container with id f2ab1ae998cc53236943895eac8d413105ed5d7e272ddfc806ad7cb75d9f4adf Sep 17 02:02:31.935201 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:31.935170 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10798-ceb822" event={"ID":"6f507459-c744-4453-80f1-58e436c623d9","Type":"ContainerStarted","Data":"2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e"} Sep 17 02:02:31.935574 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:31.935214 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10798-ceb822" event={"ID":"6f507459-c744-4453-80f1-58e436c623d9","Type":"ContainerStarted","Data":"f2ab1ae998cc53236943895eac8d413105ed5d7e272ddfc806ad7cb75d9f4adf"} Sep 17 02:02:31.935574 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:31.935455 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10798-e3aece" podUID="20a54ba2-c7ca-45e1-b193-16a7d7047faa" containerName="test-curl-10798-e3aece" containerID="cri-o://7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048" gracePeriod=2 Sep 17 02:02:31.950879 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:31.950832 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10798-ceb822" podStartSLOduration=1.9508151379999998 podStartE2EDuration="1.950815138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:02:31.948184353 +0000 UTC m=+1515.704951599" watchObservedRunningTime="2026-09-17 02:02:31.950815138 +0000 UTC m=+1515.707582383" Sep 17 02:02:32.116465 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.116436 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-47f79c" Sep 17 02:02:32.149849 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.149820 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg7c4\" (UniqueName: \"kubernetes.io/projected/6f278522-fb91-465b-b503-6761d7d1058d-kube-api-access-vg7c4\") pod \"6f278522-fb91-465b-b503-6761d7d1058d\" (UID: \"6f278522-fb91-465b-b503-6761d7d1058d\") " Sep 17 02:02:32.151979 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.151949 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f278522-fb91-465b-b503-6761d7d1058d-kube-api-access-vg7c4" (OuterVolumeSpecName: "kube-api-access-vg7c4") pod "6f278522-fb91-465b-b503-6761d7d1058d" (UID: "6f278522-fb91-465b-b503-6761d7d1058d"). InnerVolumeSpecName "kube-api-access-vg7c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:02:32.251155 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.251090 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg7c4\" (UniqueName: \"kubernetes.io/projected/6f278522-fb91-465b-b503-6761d7d1058d-kube-api-access-vg7c4\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:02:32.547222 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.547134 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10798-ceb822"] Sep 17 02:02:32.551064 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.551032 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10798-ceb822"] Sep 17 02:02:32.768690 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.768656 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f278522-fb91-465b-b503-6761d7d1058d" path="/var/lib/kubelet/pods/6f278522-fb91-465b-b503-6761d7d1058d/volumes" Sep 17 02:02:32.943366 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.943331 2586 generic.go:356] "Generic (PLEG): container finished" podID="6f278522-fb91-465b-b503-6761d7d1058d" containerID="964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3" exitCode=137 Sep 17 02:02:32.943889 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.943402 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-47f79c" Sep 17 02:02:32.943889 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.943433 2586 scope.go:117] "RemoveContainer" containerID="964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3" Sep 17 02:02:32.943889 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:02:32.943784 2586 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10798-ceb822" container="test-curl-10798-ceb822" Sep 17 02:02:32.953129 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.953111 2586 scope.go:117] "RemoveContainer" containerID="964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3" Sep 17 02:02:32.953424 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:02:32.953403 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3\": container with ID starting with 964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3 not found: ID does not exist" containerID="964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3" Sep 17 02:02:32.953476 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:32.953435 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3"} err="failed to get container status \"964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3\": rpc error: code = NotFound desc = could not find container \"964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3\": container with ID starting with 964fcd0437c52597a3466018c653b1a232da436bcbc9345142a0aae9664bbbd3 not found: ID does not exist" Sep 17 02:02:33.950669 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:33.950640 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10798-ceb822" podUID="6f507459-c744-4453-80f1-58e436c623d9" containerName="test-curl-10798-ceb822" containerID="cri-o://2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e" gracePeriod=2 Sep 17 02:02:34.124356 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:34.124321 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-e3aece" Sep 17 02:02:34.173019 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:34.172976 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9c4z\" (UniqueName: \"kubernetes.io/projected/20a54ba2-c7ca-45e1-b193-16a7d7047faa-kube-api-access-d9c4z\") pod \"20a54ba2-c7ca-45e1-b193-16a7d7047faa\" (UID: \"20a54ba2-c7ca-45e1-b193-16a7d7047faa\") " Sep 17 02:02:34.175218 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:34.175191 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a54ba2-c7ca-45e1-b193-16a7d7047faa-kube-api-access-d9c4z" (OuterVolumeSpecName: "kube-api-access-d9c4z") pod "20a54ba2-c7ca-45e1-b193-16a7d7047faa" (UID: "20a54ba2-c7ca-45e1-b193-16a7d7047faa"). InnerVolumeSpecName "kube-api-access-d9c4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:02:34.274513 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:34.274431 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9c4z\" (UniqueName: \"kubernetes.io/projected/20a54ba2-c7ca-45e1-b193-16a7d7047faa-kube-api-access-d9c4z\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:02:34.768828 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:34.768797 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a54ba2-c7ca-45e1-b193-16a7d7047faa" path="/var/lib/kubelet/pods/20a54ba2-c7ca-45e1-b193-16a7d7047faa/volumes" Sep 17 02:02:34.958044 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:34.958001 2586 generic.go:356] "Generic (PLEG): container finished" podID="20a54ba2-c7ca-45e1-b193-16a7d7047faa" containerID="7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048" exitCode=137 Sep 17 02:02:34.958044 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:34.958044 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-e3aece" Sep 17 02:02:34.958549 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:34.958109 2586 scope.go:117] "RemoveContainer" containerID="7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048" Sep 17 02:02:34.969558 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:34.969287 2586 scope.go:117] "RemoveContainer" containerID="7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048" Sep 17 02:02:34.969893 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:02:34.969857 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048\": container with ID starting with 7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048 not found: ID does not exist" containerID="7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048" Sep 17 02:02:34.969981 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:34.969902 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048"} err="failed to get container status \"7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048\": rpc error: code = NotFound desc = could not find container \"7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048\": container with ID starting with 7f4d315b8969ed7aa55a63b65fcf88782dcd60298fe90837389ba2b02ad1a048 not found: ID does not exist" Sep 17 02:02:36.136790 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:36.136755 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-ceb822" Sep 17 02:02:36.194089 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:36.194054 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9sg7\" (UniqueName: \"kubernetes.io/projected/6f507459-c744-4453-80f1-58e436c623d9-kube-api-access-q9sg7\") pod \"6f507459-c744-4453-80f1-58e436c623d9\" (UID: \"6f507459-c744-4453-80f1-58e436c623d9\") " Sep 17 02:02:36.196365 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:36.196337 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f507459-c744-4453-80f1-58e436c623d9-kube-api-access-q9sg7" (OuterVolumeSpecName: "kube-api-access-q9sg7") pod "6f507459-c744-4453-80f1-58e436c623d9" (UID: "6f507459-c744-4453-80f1-58e436c623d9"). InnerVolumeSpecName "kube-api-access-q9sg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:02:36.295292 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:36.295259 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9sg7\" (UniqueName: \"kubernetes.io/projected/6f507459-c744-4453-80f1-58e436c623d9-kube-api-access-q9sg7\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:02:36.768777 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:36.768742 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f507459-c744-4453-80f1-58e436c623d9" path="/var/lib/kubelet/pods/6f507459-c744-4453-80f1-58e436c623d9/volumes" Sep 17 02:02:36.974436 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:36.974402 2586 generic.go:356] "Generic (PLEG): container finished" podID="6f507459-c744-4453-80f1-58e436c623d9" containerID="2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e" exitCode=137 Sep 17 02:02:36.974635 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:36.974452 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10798-ceb822" Sep 17 02:02:36.974635 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:36.974462 2586 scope.go:117] "RemoveContainer" containerID="2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e" Sep 17 02:02:36.984732 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:36.984706 2586 scope.go:117] "RemoveContainer" containerID="2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e" Sep 17 02:02:36.985079 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:02:36.985056 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e\": container with ID starting with 2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e not found: ID does not exist" containerID="2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e" Sep 17 02:02:36.985179 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:36.985090 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e"} err="failed to get container status \"2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e\": rpc error: code = NotFound desc = could not find container \"2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e\": container with ID starting with 2849ace0ebe1c006f87977e196cb7f436302bf3f3e7fbfce8b49049734a1f02e not found: ID does not exist" Sep 17 02:02:41.311941 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.311859 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj"] Sep 17 02:02:41.312408 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.312390 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f507459-c744-4453-80f1-58e436c623d9" containerName="test-curl-10798-ceb822" Sep 17 02:02:41.312453 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.312410 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f507459-c744-4453-80f1-58e436c623d9" containerName="test-curl-10798-ceb822" Sep 17 02:02:41.312485 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.312480 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f507459-c744-4453-80f1-58e436c623d9" containerName="test-curl-10798-ceb822" Sep 17 02:02:41.317099 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.317077 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.319443 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.319401 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4e1c2c18-serving-cert\"" Sep 17 02:02:41.329384 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.329349 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj"] Sep 17 02:02:41.449962 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.449925 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-metrics-certs\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.450131 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.450004 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z885p\" (UniqueName: \"kubernetes.io/projected/1c1a7fc8-f070-469f-8665-69745b750204-kube-api-access-z885p\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.450131 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.450044 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-maas-api-tls\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.450131 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.450080 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1a7fc8-f070-469f-8665-69745b750204-openshift-service-ca\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.551008 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.550968 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1a7fc8-f070-469f-8665-69745b750204-openshift-service-ca\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.551186 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.551064 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-metrics-certs\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.551186 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.551102 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z885p\" (UniqueName: \"kubernetes.io/projected/1c1a7fc8-f070-469f-8665-69745b750204-kube-api-access-z885p\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.551186 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.551131 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-maas-api-tls\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.551316 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:02:41.551227 2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-serving-cert: secret "maas-api-e2e-mt-4e1c2c18-serving-cert" not found Sep 17 02:02:41.551316 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:02:41.551280 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-maas-api-tls podName:1c1a7fc8-f070-469f-8665-69745b750204 nodeName:}" failed. No retries permitted until 2026-09-17 02:02:42.051264995 +0000 UTC m=+1525.808032229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-maas-api-tls") pod "maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" (UID: "1c1a7fc8-f070-469f-8665-69745b750204") : secret "maas-api-e2e-mt-4e1c2c18-serving-cert" not found Sep 17 02:02:41.551703 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.551675 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1a7fc8-f070-469f-8665-69745b750204-openshift-service-ca\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.553666 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.553647 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-metrics-certs\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:41.564872 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:41.564810 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z885p\" (UniqueName: \"kubernetes.io/projected/1c1a7fc8-f070-469f-8665-69745b750204-kube-api-access-z885p\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:42.055442 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:42.055392 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-maas-api-tls\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:42.057922 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:42.057894 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-maas-api-tls\") pod \"maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:42.229056 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:42.229021 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:42.503071 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:42.503043 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj"] Sep 17 02:02:42.504159 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:02:42.504135 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1a7fc8_f070_469f_8665_69745b750204.slice/crio-91474b8c88c978cb37ea9f5c3059aa1a64acbf82bbd0afd8c6fea7ae5cb31cf9 WatchSource:0}: Error finding container 91474b8c88c978cb37ea9f5c3059aa1a64acbf82bbd0afd8c6fea7ae5cb31cf9: Status 404 returned error can't find the container with id 91474b8c88c978cb37ea9f5c3059aa1a64acbf82bbd0afd8c6fea7ae5cb31cf9 Sep 17 02:02:43.012983 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:43.012946 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" event={"ID":"1c1a7fc8-f070-469f-8665-69745b750204","Type":"ContainerStarted","Data":"c278152d96f45763e73a8627178875419f6362f37288cea7db591b793b30ce4f"} Sep 17 02:02:43.013159 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:43.013001 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" event={"ID":"1c1a7fc8-f070-469f-8665-69745b750204","Type":"ContainerStarted","Data":"91474b8c88c978cb37ea9f5c3059aa1a64acbf82bbd0afd8c6fea7ae5cb31cf9"} Sep 17 02:02:43.013159 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:43.013058 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:02:43.029186 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:43.029129 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" podStartSLOduration=1.6472244310000002 podStartE2EDuration="2.029110152s" totalImagesPullingTime="381.885721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:02:43.028335398 +0000 UTC m=+1526.785102653" watchObservedRunningTime="2026-09-17 02:02:43.029110152 +0000 UTC m=+1526.785877396" Sep 17 02:02:49.026006 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:02:49.025977 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:03:13.796458 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.796426 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb"] Sep 17 02:03:13.801908 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.801888 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.804904 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.804880 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4e1c2c18\"/\"kube-root-ca.crt\"" Sep 17 02:03:13.806056 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.806037 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4e1c2c18\"/\"default-dockercfg-mnkp6\"" Sep 17 02:03:13.806148 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.806076 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4e1c2c18\"/\"e2e-lifecycle-model-4e1c2c18-kserve-self-signed-certs\"" Sep 17 02:03:13.806148 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.806080 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4e1c2c18\"/\"openshift-service-ca.crt\"" Sep 17 02:03:13.822881 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.822845 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb"] Sep 17 02:03:13.871544 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.871507 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-home\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.871746 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.871552 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-tmp-dir\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.871746 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.871682 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b781b548-f480-4f86-8866-a7fbb43f617f-tls-certs\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.871746 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.871702 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-model-cache\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.871746 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.871721 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-dshm\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.871746 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.871747 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbfj\" (UniqueName: \"kubernetes.io/projected/b781b548-f480-4f86-8866-a7fbb43f617f-kube-api-access-8hbfj\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.973117 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.973081 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b781b548-f480-4f86-8866-a7fbb43f617f-tls-certs\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.973117 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.973119 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-model-cache\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.973360 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.973141 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-dshm\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.973360 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.973169 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbfj\" (UniqueName: \"kubernetes.io/projected/b781b548-f480-4f86-8866-a7fbb43f617f-kube-api-access-8hbfj\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.973360 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.973211 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-home\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.973360 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.973238 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-tmp-dir\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.973595 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.973536 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-model-cache\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.973653 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.973605 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-home\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.973698 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.973656 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-tmp-dir\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.975622 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.975599 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-dshm\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.975829 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.975811 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b781b548-f480-4f86-8866-a7fbb43f617f-tls-certs\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:13.982009 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:13.981988 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbfj\" (UniqueName: \"kubernetes.io/projected/b781b548-f480-4f86-8866-a7fbb43f617f-kube-api-access-8hbfj\") pod \"e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:14.111263 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:14.111229 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:14.387233 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:14.387204 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb"] Sep 17 02:03:14.388026 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:03:14.388004 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb781b548_f480_4f86_8866_a7fbb43f617f.slice/crio-ee79da9f3950285bdb287e6b66d0d70fb58c4e915ff5751f2935dcaf364e8310 WatchSource:0}: Error finding container ee79da9f3950285bdb287e6b66d0d70fb58c4e915ff5751f2935dcaf364e8310: Status 404 returned error can't find the container with id ee79da9f3950285bdb287e6b66d0d70fb58c4e915ff5751f2935dcaf364e8310 Sep 17 02:03:15.212763 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:15.212720 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" event={"ID":"b781b548-f480-4f86-8866-a7fbb43f617f","Type":"ContainerStarted","Data":"3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562"} Sep 17 02:03:15.213140 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:15.212773 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" event={"ID":"b781b548-f480-4f86-8866-a7fbb43f617f","Type":"ContainerStarted","Data":"ee79da9f3950285bdb287e6b66d0d70fb58c4e915ff5751f2935dcaf364e8310"} Sep 17 02:03:15.266058 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:15.266004 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" podStartSLOduration=2.265989204 podStartE2EDuration="2.265989204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:03:15.263017875 +0000 UTC m=+1559.019785120" watchObservedRunningTime="2026-09-17 02:03:15.265989204 +0000 UTC m=+1559.022756506" Sep 17 02:03:24.112239 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:24.112201 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:24.112657 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:24.112399 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:24.124817 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:24.124779 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:03:24.280228 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:03:24.280200 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:04:11.434302 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.434212 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx"] Sep 17 02:04:11.434934 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.434506 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" podUID="d2574ce8-de75-4639-9789-adbf5cc6c54d" containerName="maas-api" containerID="cri-o://8f8a2dfffa1f3ee0808f07fba9d689fd3243173bf074e8eb64d7a223d62059cc" gracePeriod=30 Sep 17 02:04:11.502012 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.501975 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn"] Sep 17 02:04:11.502572 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.502514 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" podUID="27cf525a-fc95-4c5c-a39c-16de265ba494" containerName="payload-pre-processing" containerID="cri-o://330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf" gracePeriod=30 Sep 17 02:04:11.568636 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.568570 2586 generic.go:356] "Generic (PLEG): container finished" podID="d2574ce8-de75-4639-9789-adbf5cc6c54d" containerID="8f8a2dfffa1f3ee0808f07fba9d689fd3243173bf074e8eb64d7a223d62059cc" exitCode=0 Sep 17 02:04:11.568869 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.568634 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" event={"ID":"d2574ce8-de75-4639-9789-adbf5cc6c54d","Type":"ContainerDied","Data":"8f8a2dfffa1f3ee0808f07fba9d689fd3243173bf074e8eb64d7a223d62059cc"} Sep 17 02:04:11.770894 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.770752 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 02:04:11.812134 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.812109 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 02:04:11.872480 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.872445 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2ff8\" (UniqueName: \"kubernetes.io/projected/d2574ce8-de75-4639-9789-adbf5cc6c54d-kube-api-access-v2ff8\") pod \"d2574ce8-de75-4639-9789-adbf5cc6c54d\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " Sep 17 02:04:11.872671 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.872512 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-maas-api-tls\") pod \"d2574ce8-de75-4639-9789-adbf5cc6c54d\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " Sep 17 02:04:11.872671 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.872567 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2574ce8-de75-4639-9789-adbf5cc6c54d-openshift-service-ca\") pod \"d2574ce8-de75-4639-9789-adbf5cc6c54d\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " Sep 17 02:04:11.872671 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.872654 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-metrics-certs\") pod \"d2574ce8-de75-4639-9789-adbf5cc6c54d\" (UID: \"d2574ce8-de75-4639-9789-adbf5cc6c54d\") " Sep 17 02:04:11.873022 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.872995 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2574ce8-de75-4639-9789-adbf5cc6c54d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2574ce8-de75-4639-9789-adbf5cc6c54d" (UID: "d2574ce8-de75-4639-9789-adbf5cc6c54d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:04:11.873161 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.873140 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2574ce8-de75-4639-9789-adbf5cc6c54d-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:11.874867 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.874838 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2574ce8-de75-4639-9789-adbf5cc6c54d" (UID: "d2574ce8-de75-4639-9789-adbf5cc6c54d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:04:11.875394 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.875371 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d2574ce8-de75-4639-9789-adbf5cc6c54d" (UID: "d2574ce8-de75-4639-9789-adbf5cc6c54d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:04:11.875458 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.875386 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2574ce8-de75-4639-9789-adbf5cc6c54d-kube-api-access-v2ff8" (OuterVolumeSpecName: "kube-api-access-v2ff8") pod "d2574ce8-de75-4639-9789-adbf5cc6c54d" (UID: "d2574ce8-de75-4639-9789-adbf5cc6c54d"). InnerVolumeSpecName "kube-api-access-v2ff8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:04:11.974073 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.973971 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7bf8\" (UniqueName: \"kubernetes.io/projected/27cf525a-fc95-4c5c-a39c-16de265ba494-kube-api-access-v7bf8\") pod \"27cf525a-fc95-4c5c-a39c-16de265ba494\" (UID: \"27cf525a-fc95-4c5c-a39c-16de265ba494\") " Sep 17 02:04:11.974073 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.974041 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cf525a-fc95-4c5c-a39c-16de265ba494-plugins-config-volume\") pod \"27cf525a-fc95-4c5c-a39c-16de265ba494\" (UID: \"27cf525a-fc95-4c5c-a39c-16de265ba494\") " Sep 17 02:04:11.974379 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.974359 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:11.974440 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.974375 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27cf525a-fc95-4c5c-a39c-16de265ba494-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27cf525a-fc95-4c5c-a39c-16de265ba494" (UID: "27cf525a-fc95-4c5c-a39c-16de265ba494"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:04:11.974440 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.974384 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ff8\" (UniqueName: \"kubernetes.io/projected/d2574ce8-de75-4639-9789-adbf5cc6c54d-kube-api-access-v2ff8\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:11.974440 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.974420 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2574ce8-de75-4639-9789-adbf5cc6c54d-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:11.976202 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:11.976174 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cf525a-fc95-4c5c-a39c-16de265ba494-kube-api-access-v7bf8" (OuterVolumeSpecName: "kube-api-access-v7bf8") pod "27cf525a-fc95-4c5c-a39c-16de265ba494" (UID: "27cf525a-fc95-4c5c-a39c-16de265ba494"). InnerVolumeSpecName "kube-api-access-v7bf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:04:12.075535 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.075482 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7bf8\" (UniqueName: \"kubernetes.io/projected/27cf525a-fc95-4c5c-a39c-16de265ba494-kube-api-access-v7bf8\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:12.075535 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.075524 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cf525a-fc95-4c5c-a39c-16de265ba494-plugins-config-volume\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:12.576980 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.576942 2586 generic.go:356] "Generic (PLEG): container finished" podID="27cf525a-fc95-4c5c-a39c-16de265ba494" containerID="330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf" exitCode=0 Sep 17 02:04:12.577462 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.576992 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" Sep 17 02:04:12.577462 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.577044 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" event={"ID":"27cf525a-fc95-4c5c-a39c-16de265ba494","Type":"ContainerDied","Data":"330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf"} Sep 17 02:04:12.577462 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.577108 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn" event={"ID":"27cf525a-fc95-4c5c-a39c-16de265ba494","Type":"ContainerDied","Data":"e16e30872893547f11f9281051156c012eefc99634ec68c021f7c4b296b930ed"} Sep 17 02:04:12.577462 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.577144 2586 scope.go:117] "RemoveContainer" containerID="330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf" Sep 17 02:04:12.578783 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.578763 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" Sep 17 02:04:12.578869 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.578781 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx" event={"ID":"d2574ce8-de75-4639-9789-adbf5cc6c54d","Type":"ContainerDied","Data":"7ad69aaa86e028a73bc910f0101dd5ddda32b36b30f59cde27871d2d3d6a7861"} Sep 17 02:04:12.588330 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.588312 2586 scope.go:117] "RemoveContainer" containerID="330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf" Sep 17 02:04:12.588700 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:04:12.588672 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf\": container with ID starting with 330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf not found: ID does not exist" containerID="330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf" Sep 17 02:04:12.588764 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.588713 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf"} err="failed to get container status \"330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf\": rpc error: code = NotFound desc = could not find container \"330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf\": container with ID starting with 330346318e9640b0bd3f98c4ecd6b3686bd6a8a756beac694c4d279183b1cacf not found: ID does not exist" Sep 17 02:04:12.588764 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.588737 2586 scope.go:117] "RemoveContainer" containerID="8f8a2dfffa1f3ee0808f07fba9d689fd3243173bf074e8eb64d7a223d62059cc" Sep 17 02:04:12.604817 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.604777 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx"] Sep 17 02:04:12.608395 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.608375 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3f027f-75bb9d498f-gp6wx"] Sep 17 02:04:12.618007 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.617985 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn"] Sep 17 02:04:12.621682 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.621658 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3f027f-754f94568b-2nrxn"] Sep 17 02:04:12.768844 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.768767 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27cf525a-fc95-4c5c-a39c-16de265ba494" path="/var/lib/kubelet/pods/27cf525a-fc95-4c5c-a39c-16de265ba494/volumes" Sep 17 02:04:12.777107 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:12.777076 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2574ce8-de75-4639-9789-adbf5cc6c54d" path="/var/lib/kubelet/pods/d2574ce8-de75-4639-9789-adbf5cc6c54d/volumes" Sep 17 02:04:13.387060 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.387028 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj"] Sep 17 02:04:13.387458 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.387263 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" podUID="1c1a7fc8-f070-469f-8665-69745b750204" containerName="maas-api" containerID="cri-o://c278152d96f45763e73a8627178875419f6362f37288cea7db591b793b30ce4f" gracePeriod=30 Sep 17 02:04:13.603097 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.603062 2586 generic.go:356] "Generic (PLEG): container finished" podID="1c1a7fc8-f070-469f-8665-69745b750204" containerID="c278152d96f45763e73a8627178875419f6362f37288cea7db591b793b30ce4f" exitCode=0 Sep 17 02:04:13.603542 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.603196 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" event={"ID":"1c1a7fc8-f070-469f-8665-69745b750204","Type":"ContainerDied","Data":"c278152d96f45763e73a8627178875419f6362f37288cea7db591b793b30ce4f"} Sep 17 02:04:13.683767 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.683740 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:04:13.795279 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.795249 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-maas-api-tls\") pod \"1c1a7fc8-f070-469f-8665-69745b750204\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " Sep 17 02:04:13.795496 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.795292 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-metrics-certs\") pod \"1c1a7fc8-f070-469f-8665-69745b750204\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " Sep 17 02:04:13.795496 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.795330 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1a7fc8-f070-469f-8665-69745b750204-openshift-service-ca\") pod \"1c1a7fc8-f070-469f-8665-69745b750204\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " Sep 17 02:04:13.795496 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.795434 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z885p\" (UniqueName: \"kubernetes.io/projected/1c1a7fc8-f070-469f-8665-69745b750204-kube-api-access-z885p\") pod \"1c1a7fc8-f070-469f-8665-69745b750204\" (UID: \"1c1a7fc8-f070-469f-8665-69745b750204\") " Sep 17 02:04:13.796086 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.796051 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1a7fc8-f070-469f-8665-69745b750204-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c1a7fc8-f070-469f-8665-69745b750204" (UID: "1c1a7fc8-f070-469f-8665-69745b750204"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:04:13.798157 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.798125 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c1a7fc8-f070-469f-8665-69745b750204" (UID: "1c1a7fc8-f070-469f-8665-69745b750204"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:04:13.798157 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.798141 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1c1a7fc8-f070-469f-8665-69745b750204" (UID: "1c1a7fc8-f070-469f-8665-69745b750204"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:04:13.798157 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.798148 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1a7fc8-f070-469f-8665-69745b750204-kube-api-access-z885p" (OuterVolumeSpecName: "kube-api-access-z885p") pod "1c1a7fc8-f070-469f-8665-69745b750204" (UID: "1c1a7fc8-f070-469f-8665-69745b750204"). InnerVolumeSpecName "kube-api-access-z885p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:04:13.896450 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.896413 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z885p\" (UniqueName: \"kubernetes.io/projected/1c1a7fc8-f070-469f-8665-69745b750204-kube-api-access-z885p\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:13.896450 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.896447 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:13.896450 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.896456 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c1a7fc8-f070-469f-8665-69745b750204-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:13.896788 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:13.896466 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1a7fc8-f070-469f-8665-69745b750204-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:14.615203 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:14.615163 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" event={"ID":"1c1a7fc8-f070-469f-8665-69745b750204","Type":"ContainerDied","Data":"91474b8c88c978cb37ea9f5c3059aa1a64acbf82bbd0afd8c6fea7ae5cb31cf9"} Sep 17 02:04:14.615203 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:14.615187 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj" Sep 17 02:04:14.615691 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:14.615240 2586 scope.go:117] "RemoveContainer" containerID="c278152d96f45763e73a8627178875419f6362f37288cea7db591b793b30ce4f" Sep 17 02:04:14.639958 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:14.639922 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj"] Sep 17 02:04:14.645748 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:14.645716 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e1c2c18-774fdc7df6-zhqvj"] Sep 17 02:04:14.769546 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:14.769509 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c1a7fc8-f070-469f-8665-69745b750204" path="/var/lib/kubelet/pods/1c1a7fc8-f070-469f-8665-69745b750204/volumes" Sep 17 02:04:21.898386 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:21.898351 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb"] Sep 17 02:04:36.919178 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:36.919117 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" podUID="b781b548-f480-4f86-8866-a7fbb43f617f" containerName="main" containerID="cri-o://3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562" gracePeriod=45 Sep 17 02:04:37.187252 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.187223 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:04:37.229975 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.229947 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b781b548-f480-4f86-8866-a7fbb43f617f-tls-certs\") pod \"b781b548-f480-4f86-8866-a7fbb43f617f\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " Sep 17 02:04:37.230565 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.230529 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-home\") pod \"b781b548-f480-4f86-8866-a7fbb43f617f\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " Sep 17 02:04:37.230805 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.230731 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-model-cache\") pod \"b781b548-f480-4f86-8866-a7fbb43f617f\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " Sep 17 02:04:37.230805 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.230780 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-tmp-dir\") pod \"b781b548-f480-4f86-8866-a7fbb43f617f\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " Sep 17 02:04:37.231046 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.230827 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-dshm\") pod \"b781b548-f480-4f86-8866-a7fbb43f617f\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " Sep 17 02:04:37.231046 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.230860 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hbfj\" (UniqueName: \"kubernetes.io/projected/b781b548-f480-4f86-8866-a7fbb43f617f-kube-api-access-8hbfj\") pod \"b781b548-f480-4f86-8866-a7fbb43f617f\" (UID: \"b781b548-f480-4f86-8866-a7fbb43f617f\") " Sep 17 02:04:37.231322 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.231257 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b781b548-f480-4f86-8866-a7fbb43f617f" (UID: "b781b548-f480-4f86-8866-a7fbb43f617f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:37.231529 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.231496 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-home" (OuterVolumeSpecName: "home") pod "b781b548-f480-4f86-8866-a7fbb43f617f" (UID: "b781b548-f480-4f86-8866-a7fbb43f617f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:37.231529 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.231509 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b781b548-f480-4f86-8866-a7fbb43f617f" (UID: "b781b548-f480-4f86-8866-a7fbb43f617f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:37.234333 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.234308 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b781b548-f480-4f86-8866-a7fbb43f617f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b781b548-f480-4f86-8866-a7fbb43f617f" (UID: "b781b548-f480-4f86-8866-a7fbb43f617f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:04:37.234547 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.234516 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-dshm" (OuterVolumeSpecName: "dshm") pod "b781b548-f480-4f86-8866-a7fbb43f617f" (UID: "b781b548-f480-4f86-8866-a7fbb43f617f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:37.234673 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.234645 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b781b548-f480-4f86-8866-a7fbb43f617f-kube-api-access-8hbfj" (OuterVolumeSpecName: "kube-api-access-8hbfj") pod "b781b548-f480-4f86-8866-a7fbb43f617f" (UID: "b781b548-f480-4f86-8866-a7fbb43f617f"). InnerVolumeSpecName "kube-api-access-8hbfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:04:37.332179 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.332144 2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-tmp-dir\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:37.332179 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.332179 2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-dshm\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:37.332421 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.332194 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hbfj\" (UniqueName: \"kubernetes.io/projected/b781b548-f480-4f86-8866-a7fbb43f617f-kube-api-access-8hbfj\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:37.332421 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.332208 2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b781b548-f480-4f86-8866-a7fbb43f617f-tls-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:37.332421 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.332223 2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-home\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:37.332421 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.332235 2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b781b548-f480-4f86-8866-a7fbb43f617f-model-cache\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:04:37.757279 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.757238 2586 generic.go:356] "Generic (PLEG): container finished" podID="b781b548-f480-4f86-8866-a7fbb43f617f" containerID="3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562" exitCode=0 Sep 17 02:04:37.757483 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.757284 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" Sep 17 02:04:37.757483 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.757336 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" event={"ID":"b781b548-f480-4f86-8866-a7fbb43f617f","Type":"ContainerDied","Data":"3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562"} Sep 17 02:04:37.757483 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.757388 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb" event={"ID":"b781b548-f480-4f86-8866-a7fbb43f617f","Type":"ContainerDied","Data":"ee79da9f3950285bdb287e6b66d0d70fb58c4e915ff5751f2935dcaf364e8310"} Sep 17 02:04:37.757483 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.757413 2586 scope.go:117] "RemoveContainer" containerID="3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562" Sep 17 02:04:37.768008 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.767993 2586 scope.go:117] "RemoveContainer" containerID="3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562" Sep 17 02:04:37.768270 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:04:37.768254 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562\": container with ID starting with 3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562 not found: ID does not exist" containerID="3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562" Sep 17 02:04:37.768336 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.768277 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562"} err="failed to get container status \"3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562\": rpc error: code = NotFound desc = could not find container \"3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562\": container with ID starting with 3f40bae40664e04c3b4105b7aad469e22e11d06b8ebd2871d4d4f743ddc67562 not found: ID does not exist" Sep 17 02:04:37.782766 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.782744 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb"] Sep 17 02:04:37.786260 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:37.786236 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4e1c2c18/e2e-lifecycle-model-4e1c2c18-kserve-8d498bd8c-dbrhb"] Sep 17 02:04:38.769138 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:04:38.769107 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b781b548-f480-4f86-8866-a7fbb43f617f" path="/var/lib/kubelet/pods/b781b548-f480-4f86-8866-a7fbb43f617f/volumes" Sep 17 02:05:23.818889 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:23.818853 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f"] Sep 17 02:05:23.823670 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:23.823633 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e564e1-29826840-5ts5f"] Sep 17 02:05:23.851758 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:23.851720 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd"] Sep 17 02:05:23.852012 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:23.851972 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" podUID="af2441bd-f353-48ee-ae3c-1ee3588d7f76" containerName="payload-pre-processing" containerID="cri-o://00c7846d5146c838332492a19a1d414ae5eab6f802def29f00d67cd0cffd48e8" gracePeriod=30 Sep 17 02:05:24.051332 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:24.051295 2586 generic.go:356] "Generic (PLEG): container finished" podID="af2441bd-f353-48ee-ae3c-1ee3588d7f76" containerID="00c7846d5146c838332492a19a1d414ae5eab6f802def29f00d67cd0cffd48e8" exitCode=0 Sep 17 02:05:24.051527 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:24.051371 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" event={"ID":"af2441bd-f353-48ee-ae3c-1ee3588d7f76","Type":"ContainerDied","Data":"00c7846d5146c838332492a19a1d414ae5eab6f802def29f00d67cd0cffd48e8"} Sep 17 02:05:24.108476 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:24.103812 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 02:05:24.200082 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:24.200041 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2441bd-f353-48ee-ae3c-1ee3588d7f76-plugins-config-volume\") pod \"af2441bd-f353-48ee-ae3c-1ee3588d7f76\" (UID: \"af2441bd-f353-48ee-ae3c-1ee3588d7f76\") " Sep 17 02:05:24.200283 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:24.200104 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk2sw\" (UniqueName: \"kubernetes.io/projected/af2441bd-f353-48ee-ae3c-1ee3588d7f76-kube-api-access-qk2sw\") pod \"af2441bd-f353-48ee-ae3c-1ee3588d7f76\" (UID: \"af2441bd-f353-48ee-ae3c-1ee3588d7f76\") " Sep 17 02:05:24.200504 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:24.200476 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2441bd-f353-48ee-ae3c-1ee3588d7f76-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af2441bd-f353-48ee-ae3c-1ee3588d7f76" (UID: "af2441bd-f353-48ee-ae3c-1ee3588d7f76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:05:24.202529 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:24.202499 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2441bd-f353-48ee-ae3c-1ee3588d7f76-kube-api-access-qk2sw" (OuterVolumeSpecName: "kube-api-access-qk2sw") pod "af2441bd-f353-48ee-ae3c-1ee3588d7f76" (UID: "af2441bd-f353-48ee-ae3c-1ee3588d7f76"). InnerVolumeSpecName "kube-api-access-qk2sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:05:24.302061 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:24.302024 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk2sw\" (UniqueName: \"kubernetes.io/projected/af2441bd-f353-48ee-ae3c-1ee3588d7f76-kube-api-access-qk2sw\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:05:24.302061 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:24.302056 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2441bd-f353-48ee-ae3c-1ee3588d7f76-plugins-config-volume\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:05:24.769191 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:24.769158 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eafe2171-1738-4cda-9246-1957326379d5" path="/var/lib/kubelet/pods/eafe2171-1738-4cda-9246-1957326379d5/volumes" Sep 17 02:05:25.059529 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:25.059448 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" Sep 17 02:05:25.059937 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:25.059446 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd" event={"ID":"af2441bd-f353-48ee-ae3c-1ee3588d7f76","Type":"ContainerDied","Data":"5ce6bee48d744ed8c9bdb8ae6e87729a40510e41198e1ba5759774ce584718e3"} Sep 17 02:05:25.059937 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:25.059600 2586 scope.go:117] "RemoveContainer" containerID="00c7846d5146c838332492a19a1d414ae5eab6f802def29f00d67cd0cffd48e8" Sep 17 02:05:25.076206 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:25.076180 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd"] Sep 17 02:05:25.079987 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:25.079947 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e564e1-7ddc8cb594-cwjcd"] Sep 17 02:05:26.770603 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:05:26.770556 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af2441bd-f353-48ee-ae3c-1ee3588d7f76" path="/var/lib/kubelet/pods/af2441bd-f353-48ee-ae3c-1ee3588d7f76/volumes" Sep 17 02:06:01.328545 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.328447 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10801-50f26a"] Sep 17 02:06:01.329152 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329116 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1a7fc8-f070-469f-8665-69745b750204" containerName="maas-api" Sep 17 02:06:01.329152 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329135 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1a7fc8-f070-469f-8665-69745b750204" containerName="maas-api" Sep 17 02:06:01.329276 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329157 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b781b548-f480-4f86-8866-a7fbb43f617f" containerName="main" Sep 17 02:06:01.329276 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329165 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b781b548-f480-4f86-8866-a7fbb43f617f" containerName="main" Sep 17 02:06:01.329276 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329210 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af2441bd-f353-48ee-ae3c-1ee3588d7f76" containerName="payload-pre-processing" Sep 17 02:06:01.329276 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329219 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="af2441bd-f353-48ee-ae3c-1ee3588d7f76" containerName="payload-pre-processing" Sep 17 02:06:01.329276 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329230 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2574ce8-de75-4639-9789-adbf5cc6c54d" containerName="maas-api" Sep 17 02:06:01.329276 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329238 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2574ce8-de75-4639-9789-adbf5cc6c54d" containerName="maas-api" Sep 17 02:06:01.329276 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329252 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cf525a-fc95-4c5c-a39c-16de265ba494" containerName="payload-pre-processing" Sep 17 02:06:01.329276 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329261 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cf525a-fc95-4c5c-a39c-16de265ba494" containerName="payload-pre-processing" Sep 17 02:06:01.329778 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329359 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b781b548-f480-4f86-8866-a7fbb43f617f" containerName="main" Sep 17 02:06:01.329778 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329373 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2574ce8-de75-4639-9789-adbf5cc6c54d" containerName="maas-api" Sep 17 02:06:01.329778 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329384 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="af2441bd-f353-48ee-ae3c-1ee3588d7f76" containerName="payload-pre-processing" Sep 17 02:06:01.329778 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329391 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1a7fc8-f070-469f-8665-69745b750204" containerName="maas-api" Sep 17 02:06:01.329778 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.329400 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="27cf525a-fc95-4c5c-a39c-16de265ba494" containerName="payload-pre-processing" Sep 17 02:06:01.332922 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.332902 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-50f26a" Sep 17 02:06:01.335703 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.335673 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7c2dt\"" Sep 17 02:06:01.338136 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.338111 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10801-50f26a"] Sep 17 02:06:01.477179 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.477140 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kfj\" (UniqueName: \"kubernetes.io/projected/c08d8364-3aed-4a1c-a4e7-bff687d70c67-kube-api-access-96kfj\") pod \"test-curl-10801-50f26a\" (UID: \"c08d8364-3aed-4a1c-a4e7-bff687d70c67\") " pod="opendatahub/test-curl-10801-50f26a" Sep 17 02:06:01.578666 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.578533 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96kfj\" (UniqueName: \"kubernetes.io/projected/c08d8364-3aed-4a1c-a4e7-bff687d70c67-kube-api-access-96kfj\") pod \"test-curl-10801-50f26a\" (UID: \"c08d8364-3aed-4a1c-a4e7-bff687d70c67\") " pod="opendatahub/test-curl-10801-50f26a" Sep 17 02:06:01.587271 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.587239 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kfj\" (UniqueName: \"kubernetes.io/projected/c08d8364-3aed-4a1c-a4e7-bff687d70c67-kube-api-access-96kfj\") pod \"test-curl-10801-50f26a\" (UID: \"c08d8364-3aed-4a1c-a4e7-bff687d70c67\") " pod="opendatahub/test-curl-10801-50f26a" Sep 17 02:06:01.644252 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.644216 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-50f26a" Sep 17 02:06:01.917299 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:01.917271 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10801-50f26a"] Sep 17 02:06:01.919035 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:06:01.919006 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc08d8364_3aed_4a1c_a4e7_bff687d70c67.slice/crio-56a21d645c50ca313047ef625c6b4558c9b261456718b291205b532e192cf203 WatchSource:0}: Error finding container 56a21d645c50ca313047ef625c6b4558c9b261456718b291205b532e192cf203: Status 404 returned error can't find the container with id 56a21d645c50ca313047ef625c6b4558c9b261456718b291205b532e192cf203 Sep 17 02:06:02.293980 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:02.293878 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10801-50f26a" event={"ID":"c08d8364-3aed-4a1c-a4e7-bff687d70c67","Type":"ContainerStarted","Data":"e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3"} Sep 17 02:06:02.293980 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:02.293929 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10801-50f26a" event={"ID":"c08d8364-3aed-4a1c-a4e7-bff687d70c67","Type":"ContainerStarted","Data":"56a21d645c50ca313047ef625c6b4558c9b261456718b291205b532e192cf203"} Sep 17 02:06:02.310317 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:02.310268 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10801-50f26a" podStartSLOduration=1.31025254 podStartE2EDuration="1.31025254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:06:02.307827432 +0000 UTC m=+1726.064594677" watchObservedRunningTime="2026-09-17 02:06:02.31025254 +0000 UTC m=+1726.067019784" Sep 17 02:06:02.959843 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:02.959797 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10801-50f26a"] Sep 17 02:06:02.964952 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:02.964919 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10801-50f26a"] Sep 17 02:06:03.300688 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:06:03.300656 2586 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10801-50f26a" container="test-curl-10801-50f26a" Sep 17 02:06:04.306511 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:04.306453 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10801-50f26a" podUID="c08d8364-3aed-4a1c-a4e7-bff687d70c67" containerName="test-curl-10801-50f26a" containerID="cri-o://e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3" gracePeriod=2 Sep 17 02:06:04.308792 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:06:04.308757 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10801-50f26a\" is forbidden: User \"system:node:ip-10-0-143-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-174.ec2.internal' and this object" podUID="c08d8364-3aed-4a1c-a4e7-bff687d70c67" pod="opendatahub/test-curl-10801-50f26a" Sep 17 02:06:04.361992 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:04.361949 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc"] Sep 17 02:06:04.364319 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:06:04.364284 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10801-50f26a\" is forbidden: User \"system:node:ip-10-0-143-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-174.ec2.internal' and this object" podUID="c08d8364-3aed-4a1c-a4e7-bff687d70c67" pod="opendatahub/test-curl-10801-50f26a" Sep 17 02:06:05.201198 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.201163 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq"] Sep 17 02:06:05.201682 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.201668 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08d8364-3aed-4a1c-a4e7-bff687d70c67" containerName="test-curl-10801-50f26a" Sep 17 02:06:05.201726 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.201684 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08d8364-3aed-4a1c-a4e7-bff687d70c67" containerName="test-curl-10801-50f26a" Sep 17 02:06:05.201778 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.201769 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c08d8364-3aed-4a1c-a4e7-bff687d70c67" containerName="test-curl-10801-50f26a" Sep 17 02:06:05.205215 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.205193 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.207561 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.207540 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-11a2e3-serving-cert\"" Sep 17 02:06:05.217047 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.217016 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq"] Sep 17 02:06:05.315626 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.315557 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsggn\" (UniqueName: \"kubernetes.io/projected/a9e9520f-9bc6-429d-89d5-7b3d1937c209-kube-api-access-jsggn\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.316079 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.315762 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-maas-api-tls\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.316079 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.315948 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-metrics-certs\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.316079 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.315986 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e9520f-9bc6-429d-89d5-7b3d1937c209-openshift-service-ca\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.416820 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.416781 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-metrics-certs\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.417029 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.416827 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e9520f-9bc6-429d-89d5-7b3d1937c209-openshift-service-ca\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.417029 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.416858 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsggn\" (UniqueName: \"kubernetes.io/projected/a9e9520f-9bc6-429d-89d5-7b3d1937c209-kube-api-access-jsggn\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.417029 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.416893 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-maas-api-tls\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.417559 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.417532 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e9520f-9bc6-429d-89d5-7b3d1937c209-openshift-service-ca\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.419748 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.419723 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-metrics-certs\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.419919 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.419782 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-maas-api-tls\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.426399 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.426370 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsggn\" (UniqueName: \"kubernetes.io/projected/a9e9520f-9bc6-429d-89d5-7b3d1937c209-kube-api-access-jsggn\") pod \"maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.518275 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.518188 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:05.797804 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.797770 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq"] Sep 17 02:06:05.799278 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:06:05.799255 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e9520f_9bc6_429d_89d5_7b3d1937c209.slice/crio-3409b04ab6fd3918490cd162a419b76d2e3cae58781767b097db9107983da8b5 WatchSource:0}: Error finding container 3409b04ab6fd3918490cd162a419b76d2e3cae58781767b097db9107983da8b5: Status 404 returned error can't find the container with id 3409b04ab6fd3918490cd162a419b76d2e3cae58781767b097db9107983da8b5 Sep 17 02:06:05.801142 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:05.801122 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 02:06:06.320573 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:06.320540 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" event={"ID":"a9e9520f-9bc6-429d-89d5-7b3d1937c209","Type":"ContainerStarted","Data":"24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df"} Sep 17 02:06:06.320880 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:06.320609 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" event={"ID":"a9e9520f-9bc6-429d-89d5-7b3d1937c209","Type":"ContainerStarted","Data":"3409b04ab6fd3918490cd162a419b76d2e3cae58781767b097db9107983da8b5"} Sep 17 02:06:06.321378 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:06.321358 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:06.339505 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:06.339465 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" podStartSLOduration=0.945744317 podStartE2EDuration="1.339449736s" totalImagesPullingTime="393.705419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:06:06.338541387 +0000 UTC m=+1730.095308646" watchObservedRunningTime="2026-09-17 02:06:06.339449736 +0000 UTC m=+1730.096216979" Sep 17 02:06:06.483747 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:06.483723 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-50f26a" Sep 17 02:06:06.629688 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:06.629652 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96kfj\" (UniqueName: \"kubernetes.io/projected/c08d8364-3aed-4a1c-a4e7-bff687d70c67-kube-api-access-96kfj\") pod \"c08d8364-3aed-4a1c-a4e7-bff687d70c67\" (UID: \"c08d8364-3aed-4a1c-a4e7-bff687d70c67\") " Sep 17 02:06:06.631891 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:06.631857 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08d8364-3aed-4a1c-a4e7-bff687d70c67-kube-api-access-96kfj" (OuterVolumeSpecName: "kube-api-access-96kfj") pod "c08d8364-3aed-4a1c-a4e7-bff687d70c67" (UID: "c08d8364-3aed-4a1c-a4e7-bff687d70c67"). InnerVolumeSpecName "kube-api-access-96kfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:06:06.731041 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:06.730971 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96kfj\" (UniqueName: \"kubernetes.io/projected/c08d8364-3aed-4a1c-a4e7-bff687d70c67-kube-api-access-96kfj\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:06:06.768259 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:06.768227 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c08d8364-3aed-4a1c-a4e7-bff687d70c67" path="/var/lib/kubelet/pods/c08d8364-3aed-4a1c-a4e7-bff687d70c67/volumes" Sep 17 02:06:07.327753 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:07.327717 2586 generic.go:356] "Generic (PLEG): container finished" podID="c08d8364-3aed-4a1c-a4e7-bff687d70c67" containerID="e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3" exitCode=137 Sep 17 02:06:07.328225 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:07.327760 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-50f26a" Sep 17 02:06:07.328225 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:07.327830 2586 scope.go:117] "RemoveContainer" containerID="e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3" Sep 17 02:06:07.338030 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:07.337954 2586 scope.go:117] "RemoveContainer" containerID="e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3" Sep 17 02:06:07.340146 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:06:07.338482 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3\": container with ID starting with e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3 not found: ID does not exist" containerID="e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3" Sep 17 02:06:07.340146 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:07.338530 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3"} err="failed to get container status \"e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3\": rpc error: code = NotFound desc = could not find container \"e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3\": container with ID starting with e00c06c9c9d16b660ff3a4d9f7eec4c535e8602efd3ddc16f356a0b701f6b3d3 not found: ID does not exist" Sep 17 02:06:12.333461 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:12.333430 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:06:17.022300 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:17.022268 2586 scope.go:117] "RemoveContainer" containerID="5a0e7e4cd1c7bdf70ade7eae4e0d4e41d60977ed7aacf016b5b56e37a1d30b77" Sep 17 02:06:17.031541 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:17.031520 2586 scope.go:117] "RemoveContainer" containerID="bb9a9b40986dc7b6a61cc3ab25ec42bee6a3f4809d4e0413b6c1c6e5024e51c3" Sep 17 02:06:19.382712 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.382653 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" podUID="adbb6564-5641-4114-8625-c6d058ae7ad7" containerName="main" containerID="cri-o://f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2" gracePeriod=45 Sep 17 02:06:19.644532 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.644474 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:06:19.653829 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.653809 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-model-cache\") pod \"adbb6564-5641-4114-8625-c6d058ae7ad7\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " Sep 17 02:06:19.653912 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.653850 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-home\") pod \"adbb6564-5641-4114-8625-c6d058ae7ad7\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " Sep 17 02:06:19.653963 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.653911 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbb6564-5641-4114-8625-c6d058ae7ad7-tls-certs\") pod \"adbb6564-5641-4114-8625-c6d058ae7ad7\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " Sep 17 02:06:19.653963 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.653936 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6xrm\" (UniqueName: \"kubernetes.io/projected/adbb6564-5641-4114-8625-c6d058ae7ad7-kube-api-access-h6xrm\") pod \"adbb6564-5641-4114-8625-c6d058ae7ad7\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " Sep 17 02:06:19.654071 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.654004 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-tmp-dir\") pod \"adbb6564-5641-4114-8625-c6d058ae7ad7\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " Sep 17 02:06:19.654071 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.654047 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-model-cache" (OuterVolumeSpecName: "model-cache") pod "adbb6564-5641-4114-8625-c6d058ae7ad7" (UID: "adbb6564-5641-4114-8625-c6d058ae7ad7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:06:19.654182 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.654108 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-dshm\") pod \"adbb6564-5641-4114-8625-c6d058ae7ad7\" (UID: \"adbb6564-5641-4114-8625-c6d058ae7ad7\") " Sep 17 02:06:19.654182 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.654171 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-home" (OuterVolumeSpecName: "home") pod "adbb6564-5641-4114-8625-c6d058ae7ad7" (UID: "adbb6564-5641-4114-8625-c6d058ae7ad7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:06:19.654260 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.654205 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "adbb6564-5641-4114-8625-c6d058ae7ad7" (UID: "adbb6564-5641-4114-8625-c6d058ae7ad7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:06:19.654538 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.654517 2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-tmp-dir\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:06:19.654538 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.654538 2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-model-cache\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:06:19.654914 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.654546 2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-home\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:06:19.656393 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.656369 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-dshm" (OuterVolumeSpecName: "dshm") pod "adbb6564-5641-4114-8625-c6d058ae7ad7" (UID: "adbb6564-5641-4114-8625-c6d058ae7ad7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:06:19.656722 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.656701 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbb6564-5641-4114-8625-c6d058ae7ad7-kube-api-access-h6xrm" (OuterVolumeSpecName: "kube-api-access-h6xrm") pod "adbb6564-5641-4114-8625-c6d058ae7ad7" (UID: "adbb6564-5641-4114-8625-c6d058ae7ad7"). InnerVolumeSpecName "kube-api-access-h6xrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:06:19.656722 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.656703 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adbb6564-5641-4114-8625-c6d058ae7ad7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "adbb6564-5641-4114-8625-c6d058ae7ad7" (UID: "adbb6564-5641-4114-8625-c6d058ae7ad7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:06:19.755077 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.755038 2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbb6564-5641-4114-8625-c6d058ae7ad7-tls-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:06:19.755077 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.755072 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6xrm\" (UniqueName: \"kubernetes.io/projected/adbb6564-5641-4114-8625-c6d058ae7ad7-kube-api-access-h6xrm\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:06:19.755272 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:19.755087 2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbb6564-5641-4114-8625-c6d058ae7ad7-dshm\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:06:20.417669 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:20.417635 2586 generic.go:356] "Generic (PLEG): container finished" podID="adbb6564-5641-4114-8625-c6d058ae7ad7" containerID="f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2" exitCode=0 Sep 17 02:06:20.418059 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:20.417685 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" Sep 17 02:06:20.418059 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:20.417711 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" event={"ID":"adbb6564-5641-4114-8625-c6d058ae7ad7","Type":"ContainerDied","Data":"f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2"} Sep 17 02:06:20.418059 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:20.417752 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc" event={"ID":"adbb6564-5641-4114-8625-c6d058ae7ad7","Type":"ContainerDied","Data":"a594559415441184cf3394138f4bd71eb4dbbc0beb0971bb4cd2aa0bd5191655"} Sep 17 02:06:20.418059 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:20.417768 2586 scope.go:117] "RemoveContainer" containerID="f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2" Sep 17 02:06:20.429085 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:20.429070 2586 scope.go:117] "RemoveContainer" containerID="f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2" Sep 17 02:06:20.429336 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:06:20.429319 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2\": container with ID starting with f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2 not found: ID does not exist" containerID="f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2" Sep 17 02:06:20.429406 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:20.429343 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2"} err="failed to get container status \"f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2\": rpc error: code = NotFound desc = could not find container \"f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2\": container with ID starting with f9919925b9f3cf2d08929d34c8a90dd297815c6c590541e0a2c242fa9c0334c2 not found: ID does not exist" Sep 17 02:06:20.442192 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:20.442168 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc"] Sep 17 02:06:20.446171 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:20.446150 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/auth-test-model-67a23a-kserve-6647f74f5f-wv9cc"] Sep 17 02:06:20.768326 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:20.768252 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adbb6564-5641-4114-8625-c6d058ae7ad7" path="/var/lib/kubelet/pods/adbb6564-5641-4114-8625-c6d058ae7ad7/volumes" Sep 17 02:06:40.421232 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.421195 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll"] Sep 17 02:06:40.422679 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.422648 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbb6564-5641-4114-8625-c6d058ae7ad7" containerName="main" Sep 17 02:06:40.422679 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.422674 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbb6564-5641-4114-8625-c6d058ae7ad7" containerName="main" Sep 17 02:06:40.422876 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.422862 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="adbb6564-5641-4114-8625-c6d058ae7ad7" containerName="main" Sep 17 02:06:40.428316 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.428294 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.430878 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.430854 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-320e88-serving-cert\"" Sep 17 02:06:40.435507 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.435481 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll"] Sep 17 02:06:40.550609 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.550553 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnj2z\" (UniqueName: \"kubernetes.io/projected/b217d51d-0b96-41bc-9dc9-8d5deac206ac-kube-api-access-cnj2z\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.550609 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.550613 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-metrics-certs\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.550853 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.550758 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-maas-api-tls\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.550853 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.550798 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b217d51d-0b96-41bc-9dc9-8d5deac206ac-openshift-service-ca\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.651813 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.651771 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-maas-api-tls\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.652010 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.651818 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b217d51d-0b96-41bc-9dc9-8d5deac206ac-openshift-service-ca\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.652010 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.651914 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnj2z\" (UniqueName: \"kubernetes.io/projected/b217d51d-0b96-41bc-9dc9-8d5deac206ac-kube-api-access-cnj2z\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.652010 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.651946 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-metrics-certs\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.652592 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.652541 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b217d51d-0b96-41bc-9dc9-8d5deac206ac-openshift-service-ca\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.654277 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.654253 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-maas-api-tls\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.654367 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.654301 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-metrics-certs\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.661167 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.661142 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnj2z\" (UniqueName: \"kubernetes.io/projected/b217d51d-0b96-41bc-9dc9-8d5deac206ac-kube-api-access-cnj2z\") pod \"maas-api-e2e-api-b-320e88-754988cddf-zsrll\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:40.742139 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:40.742069 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:41.021163 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:41.021137 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll"] Sep 17 02:06:41.023849 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:06:41.023809 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb217d51d_0b96_41bc_9dc9_8d5deac206ac.slice/crio-367879ff8a7b75dc504c80f11e8c9fb397dd0974f8c668db83de86a66cbf5d70 WatchSource:0}: Error finding container 367879ff8a7b75dc504c80f11e8c9fb397dd0974f8c668db83de86a66cbf5d70: Status 404 returned error can't find the container with id 367879ff8a7b75dc504c80f11e8c9fb397dd0974f8c668db83de86a66cbf5d70 Sep 17 02:06:41.550320 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:41.550276 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" event={"ID":"b217d51d-0b96-41bc-9dc9-8d5deac206ac","Type":"ContainerStarted","Data":"08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8"} Sep 17 02:06:41.550788 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:41.550345 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" event={"ID":"b217d51d-0b96-41bc-9dc9-8d5deac206ac","Type":"ContainerStarted","Data":"367879ff8a7b75dc504c80f11e8c9fb397dd0974f8c668db83de86a66cbf5d70"} Sep 17 02:06:41.550788 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:41.550455 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:06:41.569845 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:41.569791 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" podStartSLOduration=1.129501951 podStartE2EDuration="1.569772963s" totalImagesPullingTime="440.271012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:06:41.566401817 +0000 UTC m=+1765.323169088" watchObservedRunningTime="2026-09-17 02:06:41.569772963 +0000 UTC m=+1765.326540205" Sep 17 02:06:47.562477 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:06:47.562447 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:07:16.933328 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:16.933299 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 02:07:16.941162 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:16.941130 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 02:07:25.367683 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.367647 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq"] Sep 17 02:07:25.368216 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.367869 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" podUID="a9e9520f-9bc6-429d-89d5-7b3d1937c209" containerName="maas-api" containerID="cri-o://24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df" gracePeriod=30 Sep 17 02:07:25.616496 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.616469 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:07:25.695864 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.695771 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-metrics-certs\") pod \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " Sep 17 02:07:25.696029 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.695868 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsggn\" (UniqueName: \"kubernetes.io/projected/a9e9520f-9bc6-429d-89d5-7b3d1937c209-kube-api-access-jsggn\") pod \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " Sep 17 02:07:25.696029 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.695954 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-maas-api-tls\") pod \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " Sep 17 02:07:25.696029 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.695993 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e9520f-9bc6-429d-89d5-7b3d1937c209-openshift-service-ca\") pod \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\" (UID: \"a9e9520f-9bc6-429d-89d5-7b3d1937c209\") " Sep 17 02:07:25.696381 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.696355 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e9520f-9bc6-429d-89d5-7b3d1937c209-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9e9520f-9bc6-429d-89d5-7b3d1937c209" (UID: "a9e9520f-9bc6-429d-89d5-7b3d1937c209"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:07:25.698440 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.698400 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a9e9520f-9bc6-429d-89d5-7b3d1937c209" (UID: "a9e9520f-9bc6-429d-89d5-7b3d1937c209"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:07:25.698772 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.698743 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e9520f-9bc6-429d-89d5-7b3d1937c209-kube-api-access-jsggn" (OuterVolumeSpecName: "kube-api-access-jsggn") pod "a9e9520f-9bc6-429d-89d5-7b3d1937c209" (UID: "a9e9520f-9bc6-429d-89d5-7b3d1937c209"). InnerVolumeSpecName "kube-api-access-jsggn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:07:25.699005 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.698984 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9e9520f-9bc6-429d-89d5-7b3d1937c209" (UID: "a9e9520f-9bc6-429d-89d5-7b3d1937c209"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:07:25.797488 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.797452 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:07:25.797488 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.797487 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e9520f-9bc6-429d-89d5-7b3d1937c209-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:07:25.797764 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.797502 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9e9520f-9bc6-429d-89d5-7b3d1937c209-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:07:25.797764 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.797515 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsggn\" (UniqueName: \"kubernetes.io/projected/a9e9520f-9bc6-429d-89d5-7b3d1937c209-kube-api-access-jsggn\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:07:25.815260 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.815231 2586 generic.go:356] "Generic (PLEG): container finished" podID="a9e9520f-9bc6-429d-89d5-7b3d1937c209" containerID="24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df" exitCode=0 Sep 17 02:07:25.815383 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.815274 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" Sep 17 02:07:25.815383 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.815310 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" event={"ID":"a9e9520f-9bc6-429d-89d5-7b3d1937c209","Type":"ContainerDied","Data":"24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df"} Sep 17 02:07:25.815383 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.815363 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq" event={"ID":"a9e9520f-9bc6-429d-89d5-7b3d1937c209","Type":"ContainerDied","Data":"3409b04ab6fd3918490cd162a419b76d2e3cae58781767b097db9107983da8b5"} Sep 17 02:07:25.815533 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.815384 2586 scope.go:117] "RemoveContainer" containerID="24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df" Sep 17 02:07:25.828084 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.828058 2586 scope.go:117] "RemoveContainer" containerID="24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df" Sep 17 02:07:25.828517 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:07:25.828499 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df\": container with ID starting with 24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df not found: ID does not exist" containerID="24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df" Sep 17 02:07:25.828636 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.828526 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df"} err="failed to get container status \"24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df\": rpc error: code = NotFound desc = could not find container \"24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df\": container with ID starting with 24661ca830e888f4dc03663bea737617a94466835b75fe5c8fc29e54d3aa38df not found: ID does not exist" Sep 17 02:07:25.839497 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.839457 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq"] Sep 17 02:07:25.845005 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:25.844983 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-11a2e3-7677f7d65-bw4sq"] Sep 17 02:07:26.768434 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:26.768401 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e9520f-9bc6-429d-89d5-7b3d1937c209" path="/var/lib/kubelet/pods/a9e9520f-9bc6-429d-89d5-7b3d1937c209/volumes" Sep 17 02:07:36.773642 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:36.773604 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10801-1856c6"] Sep 17 02:07:36.774052 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:36.774016 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e9520f-9bc6-429d-89d5-7b3d1937c209" containerName="maas-api" Sep 17 02:07:36.774052 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:36.774027 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e9520f-9bc6-429d-89d5-7b3d1937c209" containerName="maas-api" Sep 17 02:07:36.774124 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:36.774113 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e9520f-9bc6-429d-89d5-7b3d1937c209" containerName="maas-api" Sep 17 02:07:36.778547 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:36.778521 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-1856c6" Sep 17 02:07:36.780907 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:36.780884 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7c2dt\"" Sep 17 02:07:36.781503 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:36.781478 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10801-1856c6"] Sep 17 02:07:36.801994 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:36.801966 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmvp\" (UniqueName: \"kubernetes.io/projected/1dc48edc-9486-4d17-bacf-346e909e1c21-kube-api-access-xvmvp\") pod \"test-curl-10801-1856c6\" (UID: \"1dc48edc-9486-4d17-bacf-346e909e1c21\") " pod="opendatahub/test-curl-10801-1856c6" Sep 17 02:07:36.902944 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:36.902910 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmvp\" (UniqueName: \"kubernetes.io/projected/1dc48edc-9486-4d17-bacf-346e909e1c21-kube-api-access-xvmvp\") pod \"test-curl-10801-1856c6\" (UID: \"1dc48edc-9486-4d17-bacf-346e909e1c21\") " pod="opendatahub/test-curl-10801-1856c6" Sep 17 02:07:36.911376 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:36.911349 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmvp\" (UniqueName: \"kubernetes.io/projected/1dc48edc-9486-4d17-bacf-346e909e1c21-kube-api-access-xvmvp\") pod \"test-curl-10801-1856c6\" (UID: \"1dc48edc-9486-4d17-bacf-346e909e1c21\") " pod="opendatahub/test-curl-10801-1856c6" Sep 17 02:07:37.091346 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:37.091251 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-1856c6" Sep 17 02:07:37.364047 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:37.364017 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10801-1856c6"] Sep 17 02:07:37.366100 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:07:37.366069 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dc48edc_9486_4d17_bacf_346e909e1c21.slice/crio-18e495fdc02188e4631b20fbd1578da217a8d5804fefdf02317a4766ba837dca WatchSource:0}: Error finding container 18e495fdc02188e4631b20fbd1578da217a8d5804fefdf02317a4766ba837dca: Status 404 returned error can't find the container with id 18e495fdc02188e4631b20fbd1578da217a8d5804fefdf02317a4766ba837dca Sep 17 02:07:37.898524 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:37.898485 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10801-1856c6" event={"ID":"1dc48edc-9486-4d17-bacf-346e909e1c21","Type":"ContainerStarted","Data":"150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee"} Sep 17 02:07:37.898946 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:37.898533 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10801-1856c6" event={"ID":"1dc48edc-9486-4d17-bacf-346e909e1c21","Type":"ContainerStarted","Data":"18e495fdc02188e4631b20fbd1578da217a8d5804fefdf02317a4766ba837dca"} Sep 17 02:07:37.914112 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:37.914059 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10801-1856c6" podStartSLOduration=1.914044139 podStartE2EDuration="1.914044139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:07:37.911195017 +0000 UTC m=+1821.667962262" watchObservedRunningTime="2026-09-17 02:07:37.914044139 +0000 UTC m=+1821.670811382" Sep 17 02:07:38.559391 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.559310 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10801-1856c6"] Sep 17 02:07:38.562851 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.562819 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10801-1856c6"] Sep 17 02:07:38.676079 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.676043 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10801-82bbba"] Sep 17 02:07:38.676547 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.676533 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc48edc-9486-4d17-bacf-346e909e1c21" containerName="test-curl-10801-1856c6" Sep 17 02:07:38.676630 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.676549 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc48edc-9486-4d17-bacf-346e909e1c21" containerName="test-curl-10801-1856c6" Sep 17 02:07:38.676683 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.676659 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc48edc-9486-4d17-bacf-346e909e1c21" containerName="test-curl-10801-1856c6" Sep 17 02:07:38.679844 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.679828 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-82bbba" Sep 17 02:07:38.687211 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.687181 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10801-82bbba"] Sep 17 02:07:38.723433 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.723391 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thjnp\" (UniqueName: \"kubernetes.io/projected/614a6590-94c9-40d9-9745-40006d45ff32-kube-api-access-thjnp\") pod \"test-curl-10801-82bbba\" (UID: \"614a6590-94c9-40d9-9745-40006d45ff32\") " pod="opendatahub/test-curl-10801-82bbba" Sep 17 02:07:38.824749 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.824698 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thjnp\" (UniqueName: \"kubernetes.io/projected/614a6590-94c9-40d9-9745-40006d45ff32-kube-api-access-thjnp\") pod \"test-curl-10801-82bbba\" (UID: \"614a6590-94c9-40d9-9745-40006d45ff32\") " pod="opendatahub/test-curl-10801-82bbba" Sep 17 02:07:38.832267 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.832245 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thjnp\" (UniqueName: \"kubernetes.io/projected/614a6590-94c9-40d9-9745-40006d45ff32-kube-api-access-thjnp\") pod \"test-curl-10801-82bbba\" (UID: \"614a6590-94c9-40d9-9745-40006d45ff32\") " pod="opendatahub/test-curl-10801-82bbba" Sep 17 02:07:38.904688 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:07:38.904663 2586 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10801-1856c6" container="test-curl-10801-1856c6" Sep 17 02:07:38.991077 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:38.990990 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-82bbba" Sep 17 02:07:39.266397 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:39.266372 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10801-82bbba"] Sep 17 02:07:39.268277 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:07:39.268249 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod614a6590_94c9_40d9_9745_40006d45ff32.slice/crio-b1bb71fe07f0e03c9e1c8540c156c8108f4e25344d1216cb05d35985de273d6f WatchSource:0}: Error finding container b1bb71fe07f0e03c9e1c8540c156c8108f4e25344d1216cb05d35985de273d6f: Status 404 returned error can't find the container with id b1bb71fe07f0e03c9e1c8540c156c8108f4e25344d1216cb05d35985de273d6f Sep 17 02:07:39.911795 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:39.911754 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10801-82bbba" event={"ID":"614a6590-94c9-40d9-9745-40006d45ff32","Type":"ContainerStarted","Data":"944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600"} Sep 17 02:07:39.912234 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:39.911804 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10801-82bbba" event={"ID":"614a6590-94c9-40d9-9745-40006d45ff32","Type":"ContainerStarted","Data":"b1bb71fe07f0e03c9e1c8540c156c8108f4e25344d1216cb05d35985de273d6f"} Sep 17 02:07:39.912234 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:39.911923 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10801-1856c6" podUID="1dc48edc-9486-4d17-bacf-346e909e1c21" containerName="test-curl-10801-1856c6" containerID="cri-o://150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee" gracePeriod=2 Sep 17 02:07:39.927561 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:39.927513 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10801-82bbba" podStartSLOduration=1.927497475 podStartE2EDuration="1.927497475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:07:39.926389401 +0000 UTC m=+1823.683156644" watchObservedRunningTime="2026-09-17 02:07:39.927497475 +0000 UTC m=+1823.684264718" Sep 17 02:07:40.538368 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:40.538334 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10801-82bbba"] Sep 17 02:07:40.542881 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:40.542849 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10801-82bbba"] Sep 17 02:07:40.918507 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:07:40.918476 2586 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10801-82bbba" container="test-curl-10801-82bbba" Sep 17 02:07:41.924529 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:41.924491 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10801-82bbba" podUID="614a6590-94c9-40d9-9745-40006d45ff32" containerName="test-curl-10801-82bbba" containerID="cri-o://944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600" gracePeriod=2 Sep 17 02:07:42.110354 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:42.110329 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-1856c6" Sep 17 02:07:42.153104 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:42.153073 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvmvp\" (UniqueName: \"kubernetes.io/projected/1dc48edc-9486-4d17-bacf-346e909e1c21-kube-api-access-xvmvp\") pod \"1dc48edc-9486-4d17-bacf-346e909e1c21\" (UID: \"1dc48edc-9486-4d17-bacf-346e909e1c21\") " Sep 17 02:07:42.155284 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:42.155246 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc48edc-9486-4d17-bacf-346e909e1c21-kube-api-access-xvmvp" (OuterVolumeSpecName: "kube-api-access-xvmvp") pod "1dc48edc-9486-4d17-bacf-346e909e1c21" (UID: "1dc48edc-9486-4d17-bacf-346e909e1c21"). InnerVolumeSpecName "kube-api-access-xvmvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:07:42.253864 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:42.253788 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvmvp\" (UniqueName: \"kubernetes.io/projected/1dc48edc-9486-4d17-bacf-346e909e1c21-kube-api-access-xvmvp\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:07:42.774349 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:42.774311 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc48edc-9486-4d17-bacf-346e909e1c21" path="/var/lib/kubelet/pods/1dc48edc-9486-4d17-bacf-346e909e1c21/volumes" Sep 17 02:07:42.931942 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:42.931904 2586 generic.go:356] "Generic (PLEG): container finished" podID="1dc48edc-9486-4d17-bacf-346e909e1c21" containerID="150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee" exitCode=137 Sep 17 02:07:42.932362 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:42.931965 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-1856c6" Sep 17 02:07:42.932362 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:42.932014 2586 scope.go:117] "RemoveContainer" containerID="150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee" Sep 17 02:07:42.941932 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:42.941914 2586 scope.go:117] "RemoveContainer" containerID="150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee" Sep 17 02:07:42.942198 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:07:42.942180 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee\": container with ID starting with 150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee not found: ID does not exist" containerID="150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee" Sep 17 02:07:42.942260 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:42.942208 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee"} err="failed to get container status \"150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee\": rpc error: code = NotFound desc = could not find container \"150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee\": container with ID starting with 150c850a651b8fd1a320c8531f39c41b993d5ab77bfd00e36dc2d131ebc989ee not found: ID does not exist" Sep 17 02:07:44.114802 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:44.114773 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-82bbba" Sep 17 02:07:44.169767 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:44.169735 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thjnp\" (UniqueName: \"kubernetes.io/projected/614a6590-94c9-40d9-9745-40006d45ff32-kube-api-access-thjnp\") pod \"614a6590-94c9-40d9-9745-40006d45ff32\" (UID: \"614a6590-94c9-40d9-9745-40006d45ff32\") " Sep 17 02:07:44.171936 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:44.171912 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614a6590-94c9-40d9-9745-40006d45ff32-kube-api-access-thjnp" (OuterVolumeSpecName: "kube-api-access-thjnp") pod "614a6590-94c9-40d9-9745-40006d45ff32" (UID: "614a6590-94c9-40d9-9745-40006d45ff32"). InnerVolumeSpecName "kube-api-access-thjnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:07:44.270706 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:44.270675 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thjnp\" (UniqueName: \"kubernetes.io/projected/614a6590-94c9-40d9-9745-40006d45ff32-kube-api-access-thjnp\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:07:44.770850 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:44.770817 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="614a6590-94c9-40d9-9745-40006d45ff32" path="/var/lib/kubelet/pods/614a6590-94c9-40d9-9745-40006d45ff32/volumes" Sep 17 02:07:44.946711 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:44.946678 2586 generic.go:356] "Generic (PLEG): container finished" podID="614a6590-94c9-40d9-9745-40006d45ff32" containerID="944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600" exitCode=137 Sep 17 02:07:44.946912 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:44.946722 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10801-82bbba" Sep 17 02:07:44.946912 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:44.946777 2586 scope.go:117] "RemoveContainer" containerID="944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600" Sep 17 02:07:44.955812 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:44.955793 2586 scope.go:117] "RemoveContainer" containerID="944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600" Sep 17 02:07:44.956091 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:07:44.956073 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600\": container with ID starting with 944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600 not found: ID does not exist" containerID="944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600" Sep 17 02:07:44.956138 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:44.956100 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600"} err="failed to get container status \"944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600\": rpc error: code = NotFound desc = could not find container \"944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600\": container with ID starting with 944559fbccc12da2ae26ee26c7976a1e700a13fccc47567680b48ab42e21e600 not found: ID does not exist" Sep 17 02:07:47.973267 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:47.973232 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll"] Sep 17 02:07:47.973789 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:47.973488 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" podUID="b217d51d-0b96-41bc-9dc9-8d5deac206ac" containerName="maas-api" containerID="cri-o://08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8" gracePeriod=30 Sep 17 02:07:48.231437 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.231373 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:07:48.314270 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.314235 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b217d51d-0b96-41bc-9dc9-8d5deac206ac-openshift-service-ca\") pod \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " Sep 17 02:07:48.314429 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.314300 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-maas-api-tls\") pod \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " Sep 17 02:07:48.314429 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.314328 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-metrics-certs\") pod \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " Sep 17 02:07:48.314429 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.314377 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnj2z\" (UniqueName: \"kubernetes.io/projected/b217d51d-0b96-41bc-9dc9-8d5deac206ac-kube-api-access-cnj2z\") pod \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\" (UID: \"b217d51d-0b96-41bc-9dc9-8d5deac206ac\") " Sep 17 02:07:48.314730 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.314693 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b217d51d-0b96-41bc-9dc9-8d5deac206ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b217d51d-0b96-41bc-9dc9-8d5deac206ac" (UID: "b217d51d-0b96-41bc-9dc9-8d5deac206ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:07:48.314917 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.314874 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b217d51d-0b96-41bc-9dc9-8d5deac206ac-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:07:48.317325 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.317293 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b217d51d-0b96-41bc-9dc9-8d5deac206ac" (UID: "b217d51d-0b96-41bc-9dc9-8d5deac206ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:07:48.317627 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.317564 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b217d51d-0b96-41bc-9dc9-8d5deac206ac-kube-api-access-cnj2z" (OuterVolumeSpecName: "kube-api-access-cnj2z") pod "b217d51d-0b96-41bc-9dc9-8d5deac206ac" (UID: "b217d51d-0b96-41bc-9dc9-8d5deac206ac"). InnerVolumeSpecName "kube-api-access-cnj2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:07:48.317909 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.317886 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b217d51d-0b96-41bc-9dc9-8d5deac206ac" (UID: "b217d51d-0b96-41bc-9dc9-8d5deac206ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:07:48.415509 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.415471 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:07:48.415509 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.415501 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b217d51d-0b96-41bc-9dc9-8d5deac206ac-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:07:48.415509 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.415512 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnj2z\" (UniqueName: \"kubernetes.io/projected/b217d51d-0b96-41bc-9dc9-8d5deac206ac-kube-api-access-cnj2z\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:07:48.974681 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.974648 2586 generic.go:356] "Generic (PLEG): container finished" podID="b217d51d-0b96-41bc-9dc9-8d5deac206ac" containerID="08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8" exitCode=0 Sep 17 02:07:48.975115 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.974698 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" Sep 17 02:07:48.975115 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.974742 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" event={"ID":"b217d51d-0b96-41bc-9dc9-8d5deac206ac","Type":"ContainerDied","Data":"08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8"} Sep 17 02:07:48.975115 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.974797 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll" event={"ID":"b217d51d-0b96-41bc-9dc9-8d5deac206ac","Type":"ContainerDied","Data":"367879ff8a7b75dc504c80f11e8c9fb397dd0974f8c668db83de86a66cbf5d70"} Sep 17 02:07:48.975115 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.974819 2586 scope.go:117] "RemoveContainer" containerID="08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8" Sep 17 02:07:48.984327 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.984309 2586 scope.go:117] "RemoveContainer" containerID="08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8" Sep 17 02:07:48.984620 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:07:48.984600 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8\": container with ID starting with 08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8 not found: ID does not exist" containerID="08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8" Sep 17 02:07:48.984669 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.984630 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8"} err="failed to get container status \"08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8\": rpc error: code = NotFound desc = could not find container \"08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8\": container with ID starting with 08e219441be27bdfcb4c7d108e3461eb7aa8a7e476236075607b1b967192edd8 not found: ID does not exist" Sep 17 02:07:48.992101 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.992075 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll"] Sep 17 02:07:48.995045 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:48.995027 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-320e88-754988cddf-zsrll"] Sep 17 02:07:50.767990 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:07:50.767957 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b217d51d-0b96-41bc-9dc9-8d5deac206ac" path="/var/lib/kubelet/pods/b217d51d-0b96-41bc-9dc9-8d5deac206ac/volumes" Sep 17 02:08:05.442088 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.442051 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n"] Sep 17 02:08:05.442675 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.442309 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" podUID="33e4b19c-e9e0-4d55-a915-8e2962dd822a" containerName="maas-api" containerID="cri-o://c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636" gracePeriod=30 Sep 17 02:08:05.702898 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.702835 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:08:05.793826 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.793792 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e4b19c-e9e0-4d55-a915-8e2962dd822a-openshift-service-ca\") pod \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " Sep 17 02:08:05.794027 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.793898 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-metrics-certs\") pod \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " Sep 17 02:08:05.794027 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.793939 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-maas-api-tls\") pod \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " Sep 17 02:08:05.794027 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.793999 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbw9k\" (UniqueName: \"kubernetes.io/projected/33e4b19c-e9e0-4d55-a915-8e2962dd822a-kube-api-access-bbw9k\") pod \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\" (UID: \"33e4b19c-e9e0-4d55-a915-8e2962dd822a\") " Sep 17 02:08:05.794221 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.794195 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e4b19c-e9e0-4d55-a915-8e2962dd822a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33e4b19c-e9e0-4d55-a915-8e2962dd822a" (UID: "33e4b19c-e9e0-4d55-a915-8e2962dd822a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:08:05.794427 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.794401 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e4b19c-e9e0-4d55-a915-8e2962dd822a-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:08:05.796259 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.796232 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33e4b19c-e9e0-4d55-a915-8e2962dd822a" (UID: "33e4b19c-e9e0-4d55-a915-8e2962dd822a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:08:05.796775 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.796750 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33e4b19c-e9e0-4d55-a915-8e2962dd822a" (UID: "33e4b19c-e9e0-4d55-a915-8e2962dd822a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:08:05.796775 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.796769 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e4b19c-e9e0-4d55-a915-8e2962dd822a-kube-api-access-bbw9k" (OuterVolumeSpecName: "kube-api-access-bbw9k") pod "33e4b19c-e9e0-4d55-a915-8e2962dd822a" (UID: "33e4b19c-e9e0-4d55-a915-8e2962dd822a"). InnerVolumeSpecName "kube-api-access-bbw9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:08:05.895954 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.895922 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:08:05.895954 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.895946 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e4b19c-e9e0-4d55-a915-8e2962dd822a-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:08:05.895954 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:05.895955 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbw9k\" (UniqueName: \"kubernetes.io/projected/33e4b19c-e9e0-4d55-a915-8e2962dd822a-kube-api-access-bbw9k\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:08:06.081518 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:06.081430 2586 generic.go:356] "Generic (PLEG): container finished" podID="33e4b19c-e9e0-4d55-a915-8e2962dd822a" containerID="c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636" exitCode=0 Sep 17 02:08:06.081518 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:06.081477 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" Sep 17 02:08:06.081737 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:06.081520 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" event={"ID":"33e4b19c-e9e0-4d55-a915-8e2962dd822a","Type":"ContainerDied","Data":"c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636"} Sep 17 02:08:06.081737 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:06.081572 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n" event={"ID":"33e4b19c-e9e0-4d55-a915-8e2962dd822a","Type":"ContainerDied","Data":"40719b03705eae8da091180589e055b66df08acec635dab844a91bdce84a4268"} Sep 17 02:08:06.081737 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:06.081609 2586 scope.go:117] "RemoveContainer" containerID="c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636" Sep 17 02:08:06.092562 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:06.092530 2586 scope.go:117] "RemoveContainer" containerID="c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636" Sep 17 02:08:06.092891 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:08:06.092866 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636\": container with ID starting with c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636 not found: ID does not exist" containerID="c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636" Sep 17 02:08:06.092993 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:06.092899 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636"} err="failed to get container status \"c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636\": rpc error: code = NotFound desc = could not find container \"c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636\": container with ID starting with c5247885a66487e9ff6e14d2ef17a28677ec37f6462d9fddb27b118a2a777636 not found: ID does not exist" Sep 17 02:08:06.105012 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:06.104986 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n"] Sep 17 02:08:06.110122 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:06.110100 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c5da3b-dd4fc669b-j752n"] Sep 17 02:08:06.768868 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:06.768836 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e4b19c-e9e0-4d55-a915-8e2962dd822a" path="/var/lib/kubelet/pods/33e4b19c-e9e0-4d55-a915-8e2962dd822a/volumes" Sep 17 02:08:23.543149 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.543113 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp"] Sep 17 02:08:23.543558 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.543336 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" podUID="0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23" containerName="maas-api" containerID="cri-o://018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e" gracePeriod=30 Sep 17 02:08:23.790520 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.790498 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:08:23.876703 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.876667 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-openshift-service-ca\") pod \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " Sep 17 02:08:23.876899 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.876790 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-metrics-certs\") pod \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " Sep 17 02:08:23.876899 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.876858 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-maas-api-tls\") pod \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " Sep 17 02:08:23.876899 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.876892 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hbr8\" (UniqueName: \"kubernetes.io/projected/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-kube-api-access-7hbr8\") pod \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\" (UID: \"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23\") " Sep 17 02:08:23.877057 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.876986 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23" (UID: "0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:08:23.877256 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.877234 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:08:23.879126 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.879092 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23" (UID: "0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:08:23.879126 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.879107 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23" (UID: "0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:08:23.879248 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.879154 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-kube-api-access-7hbr8" (OuterVolumeSpecName: "kube-api-access-7hbr8") pod "0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23" (UID: "0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23"). InnerVolumeSpecName "kube-api-access-7hbr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:08:23.977773 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.977739 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:08:23.977773 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.977767 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:08:23.977773 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:23.977777 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hbr8\" (UniqueName: \"kubernetes.io/projected/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23-kube-api-access-7hbr8\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:08:24.193135 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:24.193040 2586 generic.go:356] "Generic (PLEG): container finished" podID="0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23" containerID="018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e" exitCode=0 Sep 17 02:08:24.193135 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:24.193119 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" Sep 17 02:08:24.193332 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:24.193134 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" event={"ID":"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23","Type":"ContainerDied","Data":"018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e"} Sep 17 02:08:24.193332 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:24.193195 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp" event={"ID":"0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23","Type":"ContainerDied","Data":"4495ecaf128e139196bdf16e6c7fea78c975cd10a57af97e38502971a976b8d7"} Sep 17 02:08:24.193332 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:24.193225 2586 scope.go:117] "RemoveContainer" containerID="018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e" Sep 17 02:08:24.203393 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:24.203378 2586 scope.go:117] "RemoveContainer" containerID="018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e" Sep 17 02:08:24.203673 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:08:24.203651 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e\": container with ID starting with 018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e not found: ID does not exist" containerID="018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e" Sep 17 02:08:24.203734 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:24.203681 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e"} err="failed to get container status \"018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e\": rpc error: code = NotFound desc = could not find container \"018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e\": container with ID starting with 018a1c6d0f0978c4cc3ef34580e24e8ec60a7d711f7c556c1dfef6d9f747d07e not found: ID does not exist" Sep 17 02:08:24.219042 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:24.219009 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp"] Sep 17 02:08:24.222347 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:24.222321 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a3912d-6759f7cc8f-slfdp"] Sep 17 02:08:24.767863 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:08:24.767828 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23" path="/var/lib/kubelet/pods/0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23/volumes" Sep 17 02:09:00.821115 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821037 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b"] Sep 17 02:09:00.821680 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821648 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e4b19c-e9e0-4d55-a915-8e2962dd822a" containerName="maas-api" Sep 17 02:09:00.821680 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821670 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e4b19c-e9e0-4d55-a915-8e2962dd822a" containerName="maas-api" Sep 17 02:09:00.821812 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821703 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614a6590-94c9-40d9-9745-40006d45ff32" containerName="test-curl-10801-82bbba" Sep 17 02:09:00.821812 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821711 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="614a6590-94c9-40d9-9745-40006d45ff32" containerName="test-curl-10801-82bbba" Sep 17 02:09:00.821812 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821735 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b217d51d-0b96-41bc-9dc9-8d5deac206ac" containerName="maas-api" Sep 17 02:09:00.821812 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821743 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b217d51d-0b96-41bc-9dc9-8d5deac206ac" containerName="maas-api" Sep 17 02:09:00.821812 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821766 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23" containerName="maas-api" Sep 17 02:09:00.821812 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821774 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23" containerName="maas-api" Sep 17 02:09:00.822093 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821866 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="614a6590-94c9-40d9-9745-40006d45ff32" containerName="test-curl-10801-82bbba" Sep 17 02:09:00.822093 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821879 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b217d51d-0b96-41bc-9dc9-8d5deac206ac" containerName="maas-api" Sep 17 02:09:00.822093 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821887 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e4b19c-e9e0-4d55-a915-8e2962dd822a" containerName="maas-api" Sep 17 02:09:00.822093 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.821927 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d9e9b58-1ad6-42f1-b7de-9c66a62f1e23" containerName="maas-api" Sep 17 02:09:00.825535 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.825515 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:00.828140 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.828122 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0d7c34-serving-cert\"" Sep 17 02:09:00.838256 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.838231 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b"] Sep 17 02:09:00.926711 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.926664 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad84bb6-27c0-4253-b9a0-4959573ad41a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:00.926912 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.926737 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:00.926912 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.926779 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-metrics-certs\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:00.926912 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:00.926856 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbnzd\" (UniqueName: \"kubernetes.io/projected/1ad84bb6-27c0-4253-b9a0-4959573ad41a-kube-api-access-rbnzd\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:01.027711 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:01.027678 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad84bb6-27c0-4253-b9a0-4959573ad41a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:01.027907 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:01.027741 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:01.027907 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:01.027772 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-metrics-certs\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:01.028027 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:01.027901 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbnzd\" (UniqueName: \"kubernetes.io/projected/1ad84bb6-27c0-4253-b9a0-4959573ad41a-kube-api-access-rbnzd\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:01.028334 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:01.028305 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad84bb6-27c0-4253-b9a0-4959573ad41a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:01.030332 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:01.030313 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-metrics-certs\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:01.030443 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:01.030366 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:01.036548 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:01.036527 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbnzd\" (UniqueName: \"kubernetes.io/projected/1ad84bb6-27c0-4253-b9a0-4959573ad41a-kube-api-access-rbnzd\") pod \"maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:01.139954 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:01.139919 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:01.417367 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:01.417167 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b"] Sep 17 02:09:01.420803 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:09:01.420771 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad84bb6_27c0_4253_b9a0_4959573ad41a.slice/crio-005b319a59ce2ba9d60149aa09b94908dc4a0378b64f3ee66b33e6f7b2effbf1 WatchSource:0}: Error finding container 005b319a59ce2ba9d60149aa09b94908dc4a0378b64f3ee66b33e6f7b2effbf1: Status 404 returned error can't find the container with id 005b319a59ce2ba9d60149aa09b94908dc4a0378b64f3ee66b33e6f7b2effbf1 Sep 17 02:09:02.411017 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:02.410968 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" event={"ID":"1ad84bb6-27c0-4253-b9a0-4959573ad41a","Type":"ContainerStarted","Data":"bf06819553933aa614b0ec01eae015e1835667c03297ac235eb516486aee6694"} Sep 17 02:09:02.411479 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:02.411029 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:02.411479 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:02.411046 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" event={"ID":"1ad84bb6-27c0-4253-b9a0-4959573ad41a","Type":"ContainerStarted","Data":"005b319a59ce2ba9d60149aa09b94908dc4a0378b64f3ee66b33e6f7b2effbf1"} Sep 17 02:09:02.429671 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:02.429621 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" podStartSLOduration=1.790814409 podStartE2EDuration="2.429604465s" totalImagesPullingTime="638.790056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:09:02.427021923 +0000 UTC m=+1906.183789202" watchObservedRunningTime="2026-09-17 02:09:02.429604465 +0000 UTC m=+1906.186371751" Sep 17 02:09:08.422724 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:08.422696 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:09:30.521877 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.521838 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg"] Sep 17 02:09:30.525988 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.525964 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.528471 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.528445 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4ee767-serving-cert\"" Sep 17 02:09:30.536125 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.536101 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg"] Sep 17 02:09:30.606880 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.606854 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ln28\" (UniqueName: \"kubernetes.io/projected/26cfe50a-177d-4409-a6cf-7f12f1736b9e-kube-api-access-7ln28\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.607037 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.606883 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-metrics-certs\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.607037 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.606924 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cfe50a-177d-4409-a6cf-7f12f1736b9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.607037 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.607029 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.708157 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.708126 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.708336 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.708191 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ln28\" (UniqueName: \"kubernetes.io/projected/26cfe50a-177d-4409-a6cf-7f12f1736b9e-kube-api-access-7ln28\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.708336 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.708208 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-metrics-certs\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.708336 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.708240 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cfe50a-177d-4409-a6cf-7f12f1736b9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.708897 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.708877 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cfe50a-177d-4409-a6cf-7f12f1736b9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.710878 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.710842 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-metrics-certs\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.710975 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.710904 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.715794 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.715768 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ln28\" (UniqueName: \"kubernetes.io/projected/26cfe50a-177d-4409-a6cf-7f12f1736b9e-kube-api-access-7ln28\") pod \"maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:30.839315 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:30.839223 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:31.115783 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:31.115755 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg"] Sep 17 02:09:31.117664 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:09:31.117636 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cfe50a_177d_4409_a6cf_7f12f1736b9e.slice/crio-66fb305ffdef5601d17a649539ae0166505cbd7d4f16b5b0eb559437d23f30c0 WatchSource:0}: Error finding container 66fb305ffdef5601d17a649539ae0166505cbd7d4f16b5b0eb559437d23f30c0: Status 404 returned error can't find the container with id 66fb305ffdef5601d17a649539ae0166505cbd7d4f16b5b0eb559437d23f30c0 Sep 17 02:09:31.579086 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:31.579045 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" event={"ID":"26cfe50a-177d-4409-a6cf-7f12f1736b9e","Type":"ContainerStarted","Data":"51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41"} Sep 17 02:09:31.579462 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:31.579110 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:31.579462 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:31.579132 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" event={"ID":"26cfe50a-177d-4409-a6cf-7f12f1736b9e","Type":"ContainerStarted","Data":"66fb305ffdef5601d17a649539ae0166505cbd7d4f16b5b0eb559437d23f30c0"} Sep 17 02:09:31.596208 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:31.596155 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" podStartSLOduration=1.274955366 podStartE2EDuration="1.596135382s" totalImagesPullingTime="321.180016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:09:31.595842907 +0000 UTC m=+1935.352610168" watchObservedRunningTime="2026-09-17 02:09:31.596135382 +0000 UTC m=+1935.352902626" Sep 17 02:09:37.591736 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:37.591693 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:09:58.864380 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:58.864350 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-gt5rn_96d36f99-014a-474f-b1e0-74132bb0f4ac/payload-processing/0.log" Sep 17 02:09:58.997100 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:09:58.997075 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-gt5rn_96d36f99-014a-474f-b1e0-74132bb0f4ac/payload-processing/0.log" Sep 17 02:10:35.666550 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:35.666523 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-gt5rn_96d36f99-014a-474f-b1e0-74132bb0f4ac/payload-processing/0.log" Sep 17 02:10:35.790669 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:35.790636 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-gt5rn_96d36f99-014a-474f-b1e0-74132bb0f4ac/payload-processing/0.log" Sep 17 02:10:42.610042 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.610003 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf"] Sep 17 02:10:42.613812 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.613789 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.616180 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.616160 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-557e7f-serving-cert\"" Sep 17 02:10:42.620954 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.620875 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf"] Sep 17 02:10:42.728771 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.728738 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2pf\" (UniqueName: \"kubernetes.io/projected/0f6652ab-9f45-4720-b967-4581c13c5b54-kube-api-access-sl2pf\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.728978 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.728792 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.728978 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.728818 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.728978 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.728908 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6652ab-9f45-4720-b967-4581c13c5b54-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.829539 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.829500 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6652ab-9f45-4720-b967-4581c13c5b54-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.829770 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.829640 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2pf\" (UniqueName: \"kubernetes.io/projected/0f6652ab-9f45-4720-b967-4581c13c5b54-kube-api-access-sl2pf\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.829770 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.829674 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.829770 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.829698 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.829902 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:10:42.829809 2586 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-serving-cert: secret "maas-api-e2e-ipp-cleanup-557e7f-serving-cert" not found Sep 17 02:10:42.829902 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:10:42.829896 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-maas-api-tls podName:0f6652ab-9f45-4720-b967-4581c13c5b54 nodeName:}" failed. No retries permitted until 2026-09-17 02:10:43.329880205 +0000 UTC m=+2007.086647426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" (UID: "0f6652ab-9f45-4720-b967-4581c13c5b54") : secret "maas-api-e2e-ipp-cleanup-557e7f-serving-cert" not found Sep 17 02:10:42.830300 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.830275 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6652ab-9f45-4720-b967-4581c13c5b54-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.832431 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.832348 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:42.841226 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:42.841201 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2pf\" (UniqueName: \"kubernetes.io/projected/0f6652ab-9f45-4720-b967-4581c13c5b54-kube-api-access-sl2pf\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:43.334778 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:43.334738 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:43.337348 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:43.337329 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:43.528237 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:43.528190 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:43.803075 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:43.803045 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf"] Sep 17 02:10:43.804291 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:10:43.804256 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6652ab_9f45_4720_b967_4581c13c5b54.slice/crio-71ce97273ace0cc213e241026843dc6cd4b73c23bae3abc993f726360b9e87a2 WatchSource:0}: Error finding container 71ce97273ace0cc213e241026843dc6cd4b73c23bae3abc993f726360b9e87a2: Status 404 returned error can't find the container with id 71ce97273ace0cc213e241026843dc6cd4b73c23bae3abc993f726360b9e87a2 Sep 17 02:10:44.005123 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:44.005088 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" event={"ID":"0f6652ab-9f45-4720-b967-4581c13c5b54","Type":"ContainerStarted","Data":"71ce97273ace0cc213e241026843dc6cd4b73c23bae3abc993f726360b9e87a2"} Sep 17 02:10:45.012473 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:45.012432 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" event={"ID":"0f6652ab-9f45-4720-b967-4581c13c5b54","Type":"ContainerStarted","Data":"bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f"} Sep 17 02:10:45.012942 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:45.012501 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:10:45.029984 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:45.029932 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" podStartSLOduration=2.589041501 podStartE2EDuration="3.029915347s" totalImagesPullingTime="440.873846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:10:45.027738213 +0000 UTC m=+2008.784505467" watchObservedRunningTime="2026-09-17 02:10:45.029915347 +0000 UTC m=+2008.786682594" Sep 17 02:10:51.024622 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:10:51.024572 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:11:21.911747 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:21.911709 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf"] Sep 17 02:11:21.914392 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:21.911928 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" podUID="0f6652ab-9f45-4720-b967-4581c13c5b54" containerName="maas-api" containerID="cri-o://bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f" gracePeriod=30 Sep 17 02:11:22.158068 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.158039 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:11:22.229213 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.229123 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl2pf\" (UniqueName: \"kubernetes.io/projected/0f6652ab-9f45-4720-b967-4581c13c5b54-kube-api-access-sl2pf\") pod \"0f6652ab-9f45-4720-b967-4581c13c5b54\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " Sep 17 02:11:22.229213 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.229183 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-maas-api-tls\") pod \"0f6652ab-9f45-4720-b967-4581c13c5b54\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " Sep 17 02:11:22.229406 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.229272 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-metrics-certs\") pod \"0f6652ab-9f45-4720-b967-4581c13c5b54\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " Sep 17 02:11:22.229406 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.229314 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6652ab-9f45-4720-b967-4581c13c5b54-openshift-service-ca\") pod \"0f6652ab-9f45-4720-b967-4581c13c5b54\" (UID: \"0f6652ab-9f45-4720-b967-4581c13c5b54\") " Sep 17 02:11:22.229752 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.229720 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6652ab-9f45-4720-b967-4581c13c5b54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f6652ab-9f45-4720-b967-4581c13c5b54" (UID: "0f6652ab-9f45-4720-b967-4581c13c5b54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:11:22.231550 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.231523 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f6652ab-9f45-4720-b967-4581c13c5b54" (UID: "0f6652ab-9f45-4720-b967-4581c13c5b54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:11:22.231784 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.231547 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0f6652ab-9f45-4720-b967-4581c13c5b54" (UID: "0f6652ab-9f45-4720-b967-4581c13c5b54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:11:22.231784 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.231568 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6652ab-9f45-4720-b967-4581c13c5b54-kube-api-access-sl2pf" (OuterVolumeSpecName: "kube-api-access-sl2pf") pod "0f6652ab-9f45-4720-b967-4581c13c5b54" (UID: "0f6652ab-9f45-4720-b967-4581c13c5b54"). InnerVolumeSpecName "kube-api-access-sl2pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:11:22.240230 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.240159 2586 generic.go:356] "Generic (PLEG): container finished" podID="0f6652ab-9f45-4720-b967-4581c13c5b54" containerID="bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f" exitCode=0 Sep 17 02:11:22.240379 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.240345 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" Sep 17 02:11:22.240546 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.240517 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" event={"ID":"0f6652ab-9f45-4720-b967-4581c13c5b54","Type":"ContainerDied","Data":"bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f"} Sep 17 02:11:22.240668 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.240574 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf" event={"ID":"0f6652ab-9f45-4720-b967-4581c13c5b54","Type":"ContainerDied","Data":"71ce97273ace0cc213e241026843dc6cd4b73c23bae3abc993f726360b9e87a2"} Sep 17 02:11:22.240668 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.240632 2586 scope.go:117] "RemoveContainer" containerID="bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f" Sep 17 02:11:22.257819 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.257795 2586 scope.go:117] "RemoveContainer" containerID="bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f" Sep 17 02:11:22.258147 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:11:22.258129 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f\": container with ID starting with bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f not found: ID does not exist" containerID="bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f" Sep 17 02:11:22.258219 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.258155 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f"} err="failed to get container status \"bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f\": rpc error: code = NotFound desc = could not find container \"bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f\": container with ID starting with bb24d62433038eb9f1a492d7faa56449cfb2b30fd39e1e6d375548e038ca8b6f not found: ID does not exist" Sep 17 02:11:22.273696 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.273657 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf"] Sep 17 02:11:22.277105 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.277072 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-557e7f-cc45b9c9f-2zrtf"] Sep 17 02:11:22.330335 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.330302 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:11:22.330335 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.330330 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6652ab-9f45-4720-b967-4581c13c5b54-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:11:22.330335 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.330339 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl2pf\" (UniqueName: \"kubernetes.io/projected/0f6652ab-9f45-4720-b967-4581c13c5b54-kube-api-access-sl2pf\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:11:22.330569 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.330348 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6652ab-9f45-4720-b967-4581c13c5b54-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:11:22.768755 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:22.768724 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f6652ab-9f45-4720-b967-4581c13c5b54" path="/var/lib/kubelet/pods/0f6652ab-9f45-4720-b967-4581c13c5b54/volumes" Sep 17 02:11:48.225633 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.225545 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b"] Sep 17 02:11:48.226171 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.225862 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" podUID="1ad84bb6-27c0-4253-b9a0-4959573ad41a" containerName="maas-api" containerID="cri-o://bf06819553933aa614b0ec01eae015e1835667c03297ac235eb516486aee6694" gracePeriod=30 Sep 17 02:11:48.404540 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.404512 2586 generic.go:356] "Generic (PLEG): container finished" podID="1ad84bb6-27c0-4253-b9a0-4959573ad41a" containerID="bf06819553933aa614b0ec01eae015e1835667c03297ac235eb516486aee6694" exitCode=0 Sep 17 02:11:48.404696 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.404613 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" event={"ID":"1ad84bb6-27c0-4253-b9a0-4959573ad41a","Type":"ContainerDied","Data":"bf06819553933aa614b0ec01eae015e1835667c03297ac235eb516486aee6694"} Sep 17 02:11:48.502055 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.502027 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:11:48.607913 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.607872 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-metrics-certs\") pod \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " Sep 17 02:11:48.608086 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.607953 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbnzd\" (UniqueName: \"kubernetes.io/projected/1ad84bb6-27c0-4253-b9a0-4959573ad41a-kube-api-access-rbnzd\") pod \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " Sep 17 02:11:48.608086 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.608042 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad84bb6-27c0-4253-b9a0-4959573ad41a-openshift-service-ca\") pod \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " Sep 17 02:11:48.608173 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.608089 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-maas-api-tls\") pod \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\" (UID: \"1ad84bb6-27c0-4253-b9a0-4959573ad41a\") " Sep 17 02:11:48.608423 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.608397 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad84bb6-27c0-4253-b9a0-4959573ad41a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ad84bb6-27c0-4253-b9a0-4959573ad41a" (UID: "1ad84bb6-27c0-4253-b9a0-4959573ad41a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:11:48.610184 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.610147 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1ad84bb6-27c0-4253-b9a0-4959573ad41a" (UID: "1ad84bb6-27c0-4253-b9a0-4959573ad41a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:11:48.610305 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.610222 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad84bb6-27c0-4253-b9a0-4959573ad41a-kube-api-access-rbnzd" (OuterVolumeSpecName: "kube-api-access-rbnzd") pod "1ad84bb6-27c0-4253-b9a0-4959573ad41a" (UID: "1ad84bb6-27c0-4253-b9a0-4959573ad41a"). InnerVolumeSpecName "kube-api-access-rbnzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:11:48.610305 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.610290 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ad84bb6-27c0-4253-b9a0-4959573ad41a" (UID: "1ad84bb6-27c0-4253-b9a0-4959573ad41a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:11:48.709145 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.709106 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad84bb6-27c0-4253-b9a0-4959573ad41a-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:11:48.709145 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.709142 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:11:48.709145 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.709152 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad84bb6-27c0-4253-b9a0-4959573ad41a-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:11:48.709387 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:48.709161 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbnzd\" (UniqueName: \"kubernetes.io/projected/1ad84bb6-27c0-4253-b9a0-4959573ad41a-kube-api-access-rbnzd\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:11:49.412409 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:49.412377 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" Sep 17 02:11:49.412899 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:49.412415 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" event={"ID":"1ad84bb6-27c0-4253-b9a0-4959573ad41a","Type":"ContainerDied","Data":"005b319a59ce2ba9d60149aa09b94908dc4a0378b64f3ee66b33e6f7b2effbf1"} Sep 17 02:11:49.412899 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:49.412498 2586 scope.go:117] "RemoveContainer" containerID="bf06819553933aa614b0ec01eae015e1835667c03297ac235eb516486aee6694" Sep 17 02:11:49.433205 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:49.433179 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b"] Sep 17 02:11:49.437119 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:49.437089 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b"] Sep 17 02:11:50.768792 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:50.768752 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ad84bb6-27c0-4253-b9a0-4959573ad41a" path="/var/lib/kubelet/pods/1ad84bb6-27c0-4253-b9a0-4959573ad41a/volumes" Sep 17 02:11:51.418636 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:11:51.418574 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d7c34-588b8677d6-6k85b" podUID="1ad84bb6-27c0-4253-b9a0-4959573ad41a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.86:8443/health\": context deadline exceeded" Sep 17 02:12:16.979308 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:16.979268 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 02:12:16.989395 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:16.989371 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 02:12:47.732909 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:47.732870 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg"] Sep 17 02:12:47.733443 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:47.733140 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" podUID="26cfe50a-177d-4409-a6cf-7f12f1736b9e" containerName="maas-api" containerID="cri-o://51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41" gracePeriod=30 Sep 17 02:12:47.991519 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:47.991450 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:12:48.111509 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.111472 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-maas-api-tls\") pod \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " Sep 17 02:12:48.111745 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.111546 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cfe50a-177d-4409-a6cf-7f12f1736b9e-openshift-service-ca\") pod \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " Sep 17 02:12:48.111745 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.111647 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-metrics-certs\") pod \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " Sep 17 02:12:48.111745 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.111702 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ln28\" (UniqueName: \"kubernetes.io/projected/26cfe50a-177d-4409-a6cf-7f12f1736b9e-kube-api-access-7ln28\") pod \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\" (UID: \"26cfe50a-177d-4409-a6cf-7f12f1736b9e\") " Sep 17 02:12:48.111971 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.111945 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cfe50a-177d-4409-a6cf-7f12f1736b9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26cfe50a-177d-4409-a6cf-7f12f1736b9e" (UID: "26cfe50a-177d-4409-a6cf-7f12f1736b9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:12:48.113970 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.113939 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cfe50a-177d-4409-a6cf-7f12f1736b9e-kube-api-access-7ln28" (OuterVolumeSpecName: "kube-api-access-7ln28") pod "26cfe50a-177d-4409-a6cf-7f12f1736b9e" (UID: "26cfe50a-177d-4409-a6cf-7f12f1736b9e"). InnerVolumeSpecName "kube-api-access-7ln28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:12:48.113970 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.113966 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "26cfe50a-177d-4409-a6cf-7f12f1736b9e" (UID: "26cfe50a-177d-4409-a6cf-7f12f1736b9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:12:48.114196 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.113984 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26cfe50a-177d-4409-a6cf-7f12f1736b9e" (UID: "26cfe50a-177d-4409-a6cf-7f12f1736b9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:12:48.213254 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.213214 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:12:48.213254 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.213251 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ln28\" (UniqueName: \"kubernetes.io/projected/26cfe50a-177d-4409-a6cf-7f12f1736b9e-kube-api-access-7ln28\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:12:48.213254 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.213262 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26cfe50a-177d-4409-a6cf-7f12f1736b9e-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:12:48.213526 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.213272 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26cfe50a-177d-4409-a6cf-7f12f1736b9e-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:12:48.776999 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.776779 2586 generic.go:356] "Generic (PLEG): container finished" podID="26cfe50a-177d-4409-a6cf-7f12f1736b9e" containerID="51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41" exitCode=0 Sep 17 02:12:48.776999 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.776829 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" Sep 17 02:12:48.776999 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.776865 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" event={"ID":"26cfe50a-177d-4409-a6cf-7f12f1736b9e","Type":"ContainerDied","Data":"51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41"} Sep 17 02:12:48.776999 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.776918 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg" event={"ID":"26cfe50a-177d-4409-a6cf-7f12f1736b9e","Type":"ContainerDied","Data":"66fb305ffdef5601d17a649539ae0166505cbd7d4f16b5b0eb559437d23f30c0"} Sep 17 02:12:48.776999 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.776945 2586 scope.go:117] "RemoveContainer" containerID="51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41" Sep 17 02:12:48.786411 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.786390 2586 scope.go:117] "RemoveContainer" containerID="51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41" Sep 17 02:12:48.786697 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:12:48.786678 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41\": container with ID starting with 51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41 not found: ID does not exist" containerID="51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41" Sep 17 02:12:48.786774 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.786704 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41"} err="failed to get container status \"51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41\": rpc error: code = NotFound desc = could not find container \"51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41\": container with ID starting with 51c7b002d9beabce2b93acc9fbf8f0cc8e05778b815323295b258d7f91ca7d41 not found: ID does not exist" Sep 17 02:12:48.797359 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.797333 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg"] Sep 17 02:12:48.800705 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:48.800683 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ee767-5f8786848d-j7kvg"] Sep 17 02:12:50.768949 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:12:50.768918 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26cfe50a-177d-4409-a6cf-7f12f1736b9e" path="/var/lib/kubelet/pods/26cfe50a-177d-4409-a6cf-7f12f1736b9e/volumes" Sep 17 02:13:05.290536 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.290484 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5"] Sep 17 02:13:05.291368 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.291307 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" podUID="d3094a48-93ce-408f-8b9a-9b30e537c565" containerName="maas-api" containerID="cri-o://653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65" gracePeriod=30 Sep 17 02:13:05.557553 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.557487 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:13:05.693529 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.693499 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-metrics-certs\") pod \"d3094a48-93ce-408f-8b9a-9b30e537c565\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " Sep 17 02:13:05.693817 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.693628 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3094a48-93ce-408f-8b9a-9b30e537c565-openshift-service-ca\") pod \"d3094a48-93ce-408f-8b9a-9b30e537c565\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " Sep 17 02:13:05.693817 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.693664 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clkmm\" (UniqueName: \"kubernetes.io/projected/d3094a48-93ce-408f-8b9a-9b30e537c565-kube-api-access-clkmm\") pod \"d3094a48-93ce-408f-8b9a-9b30e537c565\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " Sep 17 02:13:05.693817 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.693701 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-maas-api-tls\") pod \"d3094a48-93ce-408f-8b9a-9b30e537c565\" (UID: \"d3094a48-93ce-408f-8b9a-9b30e537c565\") " Sep 17 02:13:05.694097 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.694055 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3094a48-93ce-408f-8b9a-9b30e537c565-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3094a48-93ce-408f-8b9a-9b30e537c565" (UID: "d3094a48-93ce-408f-8b9a-9b30e537c565"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:13:05.696212 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.696160 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3094a48-93ce-408f-8b9a-9b30e537c565-kube-api-access-clkmm" (OuterVolumeSpecName: "kube-api-access-clkmm") pod "d3094a48-93ce-408f-8b9a-9b30e537c565" (UID: "d3094a48-93ce-408f-8b9a-9b30e537c565"). InnerVolumeSpecName "kube-api-access-clkmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:13:05.696368 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.696261 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d3094a48-93ce-408f-8b9a-9b30e537c565" (UID: "d3094a48-93ce-408f-8b9a-9b30e537c565"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:13:05.696464 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.696424 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3094a48-93ce-408f-8b9a-9b30e537c565" (UID: "d3094a48-93ce-408f-8b9a-9b30e537c565"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:13:05.794952 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.794909 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3094a48-93ce-408f-8b9a-9b30e537c565-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:13:05.794952 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.794952 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clkmm\" (UniqueName: \"kubernetes.io/projected/d3094a48-93ce-408f-8b9a-9b30e537c565-kube-api-access-clkmm\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:13:05.795189 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.794967 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:13:05.795189 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.794981 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3094a48-93ce-408f-8b9a-9b30e537c565-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:13:05.881068 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.881023 2586 generic.go:356] "Generic (PLEG): container finished" podID="d3094a48-93ce-408f-8b9a-9b30e537c565" containerID="653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65" exitCode=0 Sep 17 02:13:05.881272 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.881073 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" Sep 17 02:13:05.881272 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.881105 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" event={"ID":"d3094a48-93ce-408f-8b9a-9b30e537c565","Type":"ContainerDied","Data":"653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65"} Sep 17 02:13:05.881272 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.881162 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5" event={"ID":"d3094a48-93ce-408f-8b9a-9b30e537c565","Type":"ContainerDied","Data":"7cb0fd3c135287155ab7f42b3d12f1aa620c41508fad27d2260b96e750b95302"} Sep 17 02:13:05.881272 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.881188 2586 scope.go:117] "RemoveContainer" containerID="653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65" Sep 17 02:13:05.891789 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.891771 2586 scope.go:117] "RemoveContainer" containerID="653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65" Sep 17 02:13:05.892068 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:13:05.892047 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65\": container with ID starting with 653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65 not found: ID does not exist" containerID="653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65" Sep 17 02:13:05.892109 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.892078 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65"} err="failed to get container status \"653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65\": rpc error: code = NotFound desc = could not find container \"653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65\": container with ID starting with 653102b9dde8558d0bc14406c33f2e1c3589f8ac7437d0b421b5192210a97b65 not found: ID does not exist" Sep 17 02:13:05.905413 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.905385 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5"] Sep 17 02:13:05.913040 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:05.913020 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-67a23a-df6c9d58c-wrnw5"] Sep 17 02:13:06.768867 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:06.768831 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3094a48-93ce-408f-8b9a-9b30e537c565" path="/var/lib/kubelet/pods/d3094a48-93ce-408f-8b9a-9b30e537c565/volumes" Sep 17 02:13:23.273277 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.273227 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt"] Sep 17 02:13:23.274059 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.273539 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" podUID="de6c603a-c0a4-4167-88a2-edcf5de51689" containerName="maas-api" containerID="cri-o://516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48" gracePeriod=30 Sep 17 02:13:23.523047 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.523021 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:13:23.566401 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.566306 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-maas-api-tls\") pod \"de6c603a-c0a4-4167-88a2-edcf5de51689\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " Sep 17 02:13:23.566401 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.566351 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6c603a-c0a4-4167-88a2-edcf5de51689-openshift-service-ca\") pod \"de6c603a-c0a4-4167-88a2-edcf5de51689\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " Sep 17 02:13:23.566401 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.566400 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-metrics-certs\") pod \"de6c603a-c0a4-4167-88a2-edcf5de51689\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " Sep 17 02:13:23.566710 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.566455 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k5zh\" (UniqueName: \"kubernetes.io/projected/de6c603a-c0a4-4167-88a2-edcf5de51689-kube-api-access-8k5zh\") pod \"de6c603a-c0a4-4167-88a2-edcf5de51689\" (UID: \"de6c603a-c0a4-4167-88a2-edcf5de51689\") " Sep 17 02:13:23.567246 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.567195 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6c603a-c0a4-4167-88a2-edcf5de51689-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de6c603a-c0a4-4167-88a2-edcf5de51689" (UID: "de6c603a-c0a4-4167-88a2-edcf5de51689"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:13:23.569885 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.569846 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de6c603a-c0a4-4167-88a2-edcf5de51689" (UID: "de6c603a-c0a4-4167-88a2-edcf5de51689"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:13:23.570015 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.569942 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "de6c603a-c0a4-4167-88a2-edcf5de51689" (UID: "de6c603a-c0a4-4167-88a2-edcf5de51689"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:13:23.570192 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.570164 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6c603a-c0a4-4167-88a2-edcf5de51689-kube-api-access-8k5zh" (OuterVolumeSpecName: "kube-api-access-8k5zh") pod "de6c603a-c0a4-4167-88a2-edcf5de51689" (UID: "de6c603a-c0a4-4167-88a2-edcf5de51689"). InnerVolumeSpecName "kube-api-access-8k5zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:13:23.667573 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.667539 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:13:23.667573 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.667567 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6c603a-c0a4-4167-88a2-edcf5de51689-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:13:23.667795 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.667612 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6c603a-c0a4-4167-88a2-edcf5de51689-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:13:23.667795 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.667622 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k5zh\" (UniqueName: \"kubernetes.io/projected/de6c603a-c0a4-4167-88a2-edcf5de51689-kube-api-access-8k5zh\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:13:23.991734 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.991690 2586 generic.go:356] "Generic (PLEG): container finished" podID="de6c603a-c0a4-4167-88a2-edcf5de51689" containerID="516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48" exitCode=0 Sep 17 02:13:23.991923 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.991740 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" Sep 17 02:13:23.991923 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.991776 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" event={"ID":"de6c603a-c0a4-4167-88a2-edcf5de51689","Type":"ContainerDied","Data":"516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48"} Sep 17 02:13:23.991923 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.991841 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt" event={"ID":"de6c603a-c0a4-4167-88a2-edcf5de51689","Type":"ContainerDied","Data":"14059e9a0635c72fd97227de1ad08df585aeaf61de5c27345fa9f4c034b4ecee"} Sep 17 02:13:23.991923 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:23.991880 2586 scope.go:117] "RemoveContainer" containerID="516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48" Sep 17 02:13:24.002754 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:24.002734 2586 scope.go:117] "RemoveContainer" containerID="516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48" Sep 17 02:13:24.003038 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:13:24.003018 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48\": container with ID starting with 516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48 not found: ID does not exist" containerID="516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48" Sep 17 02:13:24.003102 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:24.003048 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48"} err="failed to get container status \"516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48\": rpc error: code = NotFound desc = could not find container \"516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48\": container with ID starting with 516feb8b5b75784634e358220a1ded9e67c00f47a9a3fba9e47b54c2501fde48 not found: ID does not exist" Sep 17 02:13:24.016697 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:24.016672 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt"] Sep 17 02:13:24.022133 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:24.022105 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f03337-5d8b799bb6-llkkt"] Sep 17 02:13:24.768968 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:13:24.768927 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de6c603a-c0a4-4167-88a2-edcf5de51689" path="/var/lib/kubelet/pods/de6c603a-c0a4-4167-88a2-edcf5de51689/volumes" Sep 17 02:15:00.134136 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134050 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826855-zzw5v"] Sep 17 02:15:00.134654 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134531 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26cfe50a-177d-4409-a6cf-7f12f1736b9e" containerName="maas-api" Sep 17 02:15:00.134654 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134545 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="26cfe50a-177d-4409-a6cf-7f12f1736b9e" containerName="maas-api" Sep 17 02:15:00.134654 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134553 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6652ab-9f45-4720-b967-4581c13c5b54" containerName="maas-api" Sep 17 02:15:00.134654 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134559 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6652ab-9f45-4720-b967-4581c13c5b54" containerName="maas-api" Sep 17 02:15:00.134654 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134567 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad84bb6-27c0-4253-b9a0-4959573ad41a" containerName="maas-api" Sep 17 02:15:00.134654 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134573 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad84bb6-27c0-4253-b9a0-4959573ad41a" containerName="maas-api" Sep 17 02:15:00.134654 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134619 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3094a48-93ce-408f-8b9a-9b30e537c565" containerName="maas-api" Sep 17 02:15:00.134654 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134628 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3094a48-93ce-408f-8b9a-9b30e537c565" containerName="maas-api" Sep 17 02:15:00.134654 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134636 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de6c603a-c0a4-4167-88a2-edcf5de51689" containerName="maas-api" Sep 17 02:15:00.134654 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134642 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="de6c603a-c0a4-4167-88a2-edcf5de51689" containerName="maas-api" Sep 17 02:15:00.134977 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134717 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6652ab-9f45-4720-b967-4581c13c5b54" containerName="maas-api" Sep 17 02:15:00.134977 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134726 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3094a48-93ce-408f-8b9a-9b30e537c565" containerName="maas-api" Sep 17 02:15:00.134977 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134733 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="de6c603a-c0a4-4167-88a2-edcf5de51689" containerName="maas-api" Sep 17 02:15:00.134977 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134740 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad84bb6-27c0-4253-b9a0-4959573ad41a" containerName="maas-api" Sep 17 02:15:00.134977 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.134746 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="26cfe50a-177d-4409-a6cf-7f12f1736b9e" containerName="maas-api" Sep 17 02:15:00.142285 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.140406 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826855-zzw5v" Sep 17 02:15:00.144734 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.143316 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:15:00.147241 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.147202 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826855-zzw5v"] Sep 17 02:15:00.156780 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.156754 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826855-zzw5v" Sep 17 02:15:00.432485 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.432452 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826855-zzw5v"] Sep 17 02:15:00.433933 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:15:00.433907 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093e15cc_53b4_4442_8dab_dc43b62ac89d.slice/crio-c8b69f813ad981add9e102a781f3de917d9db270e3c7ec8bab9d8b89c5f74436 WatchSource:0}: Error finding container c8b69f813ad981add9e102a781f3de917d9db270e3c7ec8bab9d8b89c5f74436: Status 404 returned error can't find the container with id c8b69f813ad981add9e102a781f3de917d9db270e3c7ec8bab9d8b89c5f74436 Sep 17 02:15:00.555334 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:00.555298 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826855-zzw5v" event={"ID":"093e15cc-53b4-4442-8dab-dc43b62ac89d","Type":"ContainerStarted","Data":"c8b69f813ad981add9e102a781f3de917d9db270e3c7ec8bab9d8b89c5f74436"} Sep 17 02:15:01.562909 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:01.562876 2586 generic.go:356] "Generic (PLEG): container finished" podID="093e15cc-53b4-4442-8dab-dc43b62ac89d" containerID="b4dfbf91a85673c59cf71fada99e59421df8a226f2382034c31f6110619e9e48" exitCode=0 Sep 17 02:15:01.563305 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:01.562961 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826855-zzw5v" event={"ID":"093e15cc-53b4-4442-8dab-dc43b62ac89d","Type":"ContainerDied","Data":"b4dfbf91a85673c59cf71fada99e59421df8a226f2382034c31f6110619e9e48"} Sep 17 02:15:02.706088 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:02.706060 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826855-zzw5v" Sep 17 02:15:03.575859 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:03.575828 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826855-zzw5v" Sep 17 02:15:03.575859 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:03.575854 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826855-zzw5v" event={"ID":"093e15cc-53b4-4442-8dab-dc43b62ac89d","Type":"ContainerDied","Data":"c8b69f813ad981add9e102a781f3de917d9db270e3c7ec8bab9d8b89c5f74436"} Sep 17 02:15:03.576115 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:15:03.575902 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b69f813ad981add9e102a781f3de917d9db270e3c7ec8bab9d8b89c5f74436" Sep 17 02:17:17.022952 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:17:17.022923 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 02:17:17.033443 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:17:17.033418 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 02:18:38.237528 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.237484 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq"] Sep 17 02:18:38.238034 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.237979 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093e15cc-53b4-4442-8dab-dc43b62ac89d" containerName="cleanup" Sep 17 02:18:38.238034 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.237993 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="093e15cc-53b4-4442-8dab-dc43b62ac89d" containerName="cleanup" Sep 17 02:18:38.238108 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.238080 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="093e15cc-53b4-4442-8dab-dc43b62ac89d" containerName="cleanup" Sep 17 02:18:38.241282 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.241263 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.244692 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.244670 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:18:38.244885 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.244704 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 02:18:38.245004 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.244754 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 02:18:38.245004 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.244757 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kpnw8\"" Sep 17 02:18:38.245125 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.244798 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 17 02:18:38.249254 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.249232 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq"] Sep 17 02:18:38.351819 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.351779 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-openshift-service-ca\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.351819 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.351818 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-maas-api-tls\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.352049 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.351837 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-metrics-certs\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.352049 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.351939 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd78f\" (UniqueName: \"kubernetes.io/projected/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-kube-api-access-nd78f\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.453055 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.453016 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd78f\" (UniqueName: \"kubernetes.io/projected/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-kube-api-access-nd78f\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.453241 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.453116 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-openshift-service-ca\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.453241 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.453136 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-maas-api-tls\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.453241 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.453156 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-metrics-certs\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.453819 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.453793 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-openshift-service-ca\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.455731 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.455714 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-metrics-certs\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.455894 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.455873 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-maas-api-tls\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.460916 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.460892 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd78f\" (UniqueName: \"kubernetes.io/projected/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-kube-api-access-nd78f\") pod \"maas-api-59f6566bbc-7ffpq\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.554270 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.554178 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:38.827754 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.827726 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq"] Sep 17 02:18:38.829187 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:18:38.829160 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee463b4a_4ea0_4904_8cc2_4a3f5b2828c4.slice/crio-da4d8b33162670587831e29b25282283ba53d8dee57945c04a2b5df685a9d7f6 WatchSource:0}: Error finding container da4d8b33162670587831e29b25282283ba53d8dee57945c04a2b5df685a9d7f6: Status 404 returned error can't find the container with id da4d8b33162670587831e29b25282283ba53d8dee57945c04a2b5df685a9d7f6 Sep 17 02:18:38.830523 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.830506 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 02:18:38.852377 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:38.852344 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" event={"ID":"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4","Type":"ContainerStarted","Data":"da4d8b33162670587831e29b25282283ba53d8dee57945c04a2b5df685a9d7f6"} Sep 17 02:18:39.861625 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:39.861572 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" event={"ID":"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4","Type":"ContainerStarted","Data":"9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd"} Sep 17 02:18:39.862129 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:39.861659 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:39.879938 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:39.879878 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" podStartSLOduration=1.4887617 podStartE2EDuration="1.879857686s" totalImagesPullingTime="391.095986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:18:39.87764202 +0000 UTC m=+2483.634409295" watchObservedRunningTime="2026-09-17 02:18:39.879857686 +0000 UTC m=+2483.636624931" Sep 17 02:18:45.873786 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:45.873754 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:46.108087 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.108049 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn"] Sep 17 02:18:46.111857 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.111833 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.119662 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.119633 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn"] Sep 17 02:18:46.237451 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.237359 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3a99460-d5c0-4260-95d9-c6a9395dde1a-maas-api-tls\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.237657 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.237467 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a99460-d5c0-4260-95d9-c6a9395dde1a-openshift-service-ca\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.237657 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.237508 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a99460-d5c0-4260-95d9-c6a9395dde1a-metrics-certs\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.237657 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.237535 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxpb\" (UniqueName: \"kubernetes.io/projected/a3a99460-d5c0-4260-95d9-c6a9395dde1a-kube-api-access-6bxpb\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.338255 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.338218 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a99460-d5c0-4260-95d9-c6a9395dde1a-openshift-service-ca\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.338424 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.338264 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a99460-d5c0-4260-95d9-c6a9395dde1a-metrics-certs\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.338424 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.338288 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxpb\" (UniqueName: \"kubernetes.io/projected/a3a99460-d5c0-4260-95d9-c6a9395dde1a-kube-api-access-6bxpb\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.338424 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.338329 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3a99460-d5c0-4260-95d9-c6a9395dde1a-maas-api-tls\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.339001 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.338976 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3a99460-d5c0-4260-95d9-c6a9395dde1a-openshift-service-ca\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.341001 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.340981 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3a99460-d5c0-4260-95d9-c6a9395dde1a-metrics-certs\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.341066 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.341052 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3a99460-d5c0-4260-95d9-c6a9395dde1a-maas-api-tls\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.348219 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.348192 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxpb\" (UniqueName: \"kubernetes.io/projected/a3a99460-d5c0-4260-95d9-c6a9395dde1a-kube-api-access-6bxpb\") pod \"maas-api-86d9ddc875-rpwjn\" (UID: \"a3a99460-d5c0-4260-95d9-c6a9395dde1a\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.425855 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.425823 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:46.702995 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.702964 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn"] Sep 17 02:18:46.704046 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:18:46.704018 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a99460_d5c0_4260_95d9_c6a9395dde1a.slice/crio-2565a59c114b642a3b8f422313568e6e4a69de3c7e0c816cbaf0074ef6f06b9e WatchSource:0}: Error finding container 2565a59c114b642a3b8f422313568e6e4a69de3c7e0c816cbaf0074ef6f06b9e: Status 404 returned error can't find the container with id 2565a59c114b642a3b8f422313568e6e4a69de3c7e0c816cbaf0074ef6f06b9e Sep 17 02:18:46.903439 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:46.903405 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" event={"ID":"a3a99460-d5c0-4260-95d9-c6a9395dde1a","Type":"ContainerStarted","Data":"2565a59c114b642a3b8f422313568e6e4a69de3c7e0c816cbaf0074ef6f06b9e"} Sep 17 02:18:47.917145 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:47.917104 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" event={"ID":"a3a99460-d5c0-4260-95d9-c6a9395dde1a","Type":"ContainerStarted","Data":"593415e8d9541491e54afcd5ff1d21c0f21f653c141ad1dfd7b3341dde3b7b8a"} Sep 17 02:18:47.917536 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:47.917273 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:47.933976 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:47.933929 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" podStartSLOduration=1.555725993 podStartE2EDuration="1.933915411s" totalImagesPullingTime="378.189418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:18:47.933002947 +0000 UTC m=+2491.689770216" watchObservedRunningTime="2026-09-17 02:18:47.933915411 +0000 UTC m=+2491.690682658" Sep 17 02:18:53.929337 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:53.929300 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-rpwjn" Sep 17 02:18:53.979959 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:53.979923 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq"] Sep 17 02:18:53.980203 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:53.980177 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" podUID="ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4" containerName="maas-api" containerID="cri-o://9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd" gracePeriod=30 Sep 17 02:18:54.223126 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.223099 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:54.315398 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.315352 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-maas-api-tls\") pod \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " Sep 17 02:18:54.315627 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.315407 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-openshift-service-ca\") pod \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " Sep 17 02:18:54.315627 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.315474 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-metrics-certs\") pod \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " Sep 17 02:18:54.315627 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.315497 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd78f\" (UniqueName: \"kubernetes.io/projected/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-kube-api-access-nd78f\") pod \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\" (UID: \"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4\") " Sep 17 02:18:54.315825 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.315797 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4" (UID: "ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:18:54.317803 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.317776 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4" (UID: "ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:18:54.317803 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.317788 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4" (UID: "ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:18:54.317939 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.317924 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-kube-api-access-nd78f" (OuterVolumeSpecName: "kube-api-access-nd78f") pod "ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4" (UID: "ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4"). InnerVolumeSpecName "kube-api-access-nd78f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:18:54.417212 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.417159 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-maas-api-tls\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:18:54.417212 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.417203 2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-openshift-service-ca\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:18:54.417212 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.417213 2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-metrics-certs\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:18:54.417212 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.417223 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd78f\" (UniqueName: \"kubernetes.io/projected/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4-kube-api-access-nd78f\") on node \"ip-10-0-143-174.ec2.internal\" DevicePath \"\"" Sep 17 02:18:54.958780 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.958744 2586 generic.go:356] "Generic (PLEG): container finished" podID="ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4" containerID="9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd" exitCode=0 Sep 17 02:18:54.959193 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.958788 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" Sep 17 02:18:54.959193 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.958823 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" event={"ID":"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4","Type":"ContainerDied","Data":"9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd"} Sep 17 02:18:54.959193 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.958875 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq" event={"ID":"ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4","Type":"ContainerDied","Data":"da4d8b33162670587831e29b25282283ba53d8dee57945c04a2b5df685a9d7f6"} Sep 17 02:18:54.959193 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.958897 2586 scope.go:117] "RemoveContainer" containerID="9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd" Sep 17 02:18:54.968532 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.968514 2586 scope.go:117] "RemoveContainer" containerID="9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd" Sep 17 02:18:54.968802 ip-10-0-143-174 kubenswrapper[2586]: E0917 02:18:54.968781 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd\": container with ID starting with 9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd not found: ID does not exist" containerID="9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd" Sep 17 02:18:54.968860 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.968811 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd"} err="failed to get container status \"9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd\": rpc error: code = NotFound desc = could not find container \"9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd\": container with ID starting with 9954b738e70b348eee03b47c6f09d8a665eb1e751733de2ae18b40885c3c90fd not found: ID does not exist" Sep 17 02:18:54.976154 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.976096 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq"] Sep 17 02:18:54.979648 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:54.979626 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-7ffpq"] Sep 17 02:18:56.769246 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:18:56.769210 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4" path="/var/lib/kubelet/pods/ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4/volumes" Sep 17 02:19:09.180210 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:09.180123 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f62th_31ad03fc-51f4-46b5-9a17-4d7f38513e84/global-pull-secret-syncer/0.log" Sep 17 02:19:09.329436 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:09.329406 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wvqr2_6e7731ab-68de-44b8-8897-daeee9096d24/konnectivity-agent/0.log" Sep 17 02:19:09.371194 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:09.371165 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-174.ec2.internal_65ada5008fdb1ae2d02491334493ce88/haproxy/0.log" Sep 17 02:19:14.055290 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:14.055256 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68cbd6f44c-tb4zm_31aa68d9-e388-4cd2-99f2-71c02f71ade6/authorino/0.log" Sep 17 02:19:14.153896 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:14.153865 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-pzjmm_1d8b47d2-d6ac-4e7f-a157-df5e9112a150/kuadrant-console-plugin/0.log" Sep 17 02:19:15.628784 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.628751 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad3959eb-40b8-4416-aace-ab97d641a5ba/alertmanager/0.log" Sep 17 02:19:15.658761 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.658735 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad3959eb-40b8-4416-aace-ab97d641a5ba/config-reloader/0.log" Sep 17 02:19:15.678222 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.678197 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad3959eb-40b8-4416-aace-ab97d641a5ba/kube-rbac-proxy-web/0.log" Sep 17 02:19:15.697221 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.697196 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad3959eb-40b8-4416-aace-ab97d641a5ba/kube-rbac-proxy/0.log" Sep 17 02:19:15.719853 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.719829 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad3959eb-40b8-4416-aace-ab97d641a5ba/kube-rbac-proxy-metric/0.log" Sep 17 02:19:15.739943 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.739921 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad3959eb-40b8-4416-aace-ab97d641a5ba/prom-label-proxy/0.log" Sep 17 02:19:15.759983 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.759964 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad3959eb-40b8-4416-aace-ab97d641a5ba/init-config-reloader/0.log" Sep 17 02:19:15.794379 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.794342 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-mp6xs_d637b01d-51f5-4846-a45e-6968f5411206/cluster-monitoring-operator/0.log" Sep 17 02:19:15.817276 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.817247 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-vqz77_deab9266-d7c5-4886-a2b5-6691569a4134/kube-state-metrics/0.log" Sep 17 02:19:15.835400 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.835371 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-vqz77_deab9266-d7c5-4886-a2b5-6691569a4134/kube-rbac-proxy-main/0.log" Sep 17 02:19:15.853785 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.853767 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-vqz77_deab9266-d7c5-4886-a2b5-6691569a4134/kube-rbac-proxy-self/0.log" Sep 17 02:19:15.878922 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.878864 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bc54fcff7-8c987_8e594e9e-2526-41cd-9a11-c44e0dca792c/metrics-server/0.log" Sep 17 02:19:15.941223 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.941194 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4ckfg_0950289d-133e-441f-9ea8-ccc434f1c101/node-exporter/0.log" Sep 17 02:19:15.959901 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.959876 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4ckfg_0950289d-133e-441f-9ea8-ccc434f1c101/kube-rbac-proxy/0.log" Sep 17 02:19:15.980801 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:15.980779 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4ckfg_0950289d-133e-441f-9ea8-ccc434f1c101/init-textfile/0.log" Sep 17 02:19:16.164739 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:16.164657 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-drs75_7b37e713-d8f4-4a88-9c98-414a2c310553/kube-rbac-proxy-main/0.log" Sep 17 02:19:16.184178 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:16.184148 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-drs75_7b37e713-d8f4-4a88-9c98-414a2c310553/kube-rbac-proxy-self/0.log" Sep 17 02:19:16.203065 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:16.203038 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-drs75_7b37e713-d8f4-4a88-9c98-414a2c310553/openshift-state-metrics/0.log" Sep 17 02:19:16.527444 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:16.527353 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cf5cf6d8c-k68vt_7322fa5a-a596-48a9-b0fe-3d6b76cae3ab/telemeter-client/0.log" Sep 17 02:19:16.546185 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:16.546156 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cf5cf6d8c-k68vt_7322fa5a-a596-48a9-b0fe-3d6b76cae3ab/reload/0.log" Sep 17 02:19:16.567626 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:16.567574 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cf5cf6d8c-k68vt_7322fa5a-a596-48a9-b0fe-3d6b76cae3ab/kube-rbac-proxy/0.log" Sep 17 02:19:18.336982 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:18.336949 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-9r2gb_5467c639-5928-4a5f-a109-18c20b43fcfa/networking-console-plugin/0.log" Sep 17 02:19:18.966602 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:18.966550 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/2.log" Sep 17 02:19:18.971812 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:18.971792 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xz5g9_d084eabd-5582-43e0-84bf-d5a860eb8092/console-operator/3.log" Sep 17 02:19:19.818820 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.818785 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr"] Sep 17 02:19:19.819359 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.819277 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4" containerName="maas-api" Sep 17 02:19:19.819359 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.819295 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4" containerName="maas-api" Sep 17 02:19:19.819495 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.819426 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee463b4a-4ea0-4904-8cc2-4a3f5b2828c4" containerName="maas-api" Sep 17 02:19:19.823976 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.823959 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:19.826519 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.826502 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nlq6t\"/\"kube-root-ca.crt\"" Sep 17 02:19:19.826648 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.826516 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nlq6t\"/\"default-dockercfg-cl7v2\"" Sep 17 02:19:19.827668 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.827650 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nlq6t\"/\"openshift-service-ca.crt\"" Sep 17 02:19:19.831482 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.831462 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr"] Sep 17 02:19:19.978343 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.978310 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-podres\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:19.978514 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.978363 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-proc\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:19.978514 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.978382 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcq2g\" (UniqueName: \"kubernetes.io/projected/6f30c289-8fda-4a5a-a786-68897460562e-kube-api-access-qcq2g\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:19.978514 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.978477 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-sys\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:19.978648 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:19.978522 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-lib-modules\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.079244 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.079160 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-lib-modules\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.079244 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.079223 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-podres\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.079432 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.079262 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-proc\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.079432 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.079281 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcq2g\" (UniqueName: \"kubernetes.io/projected/6f30c289-8fda-4a5a-a786-68897460562e-kube-api-access-qcq2g\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.079432 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.079342 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-lib-modules\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.079432 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.079347 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-sys\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.079432 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.079371 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-proc\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.079432 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.079397 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-sys\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.079432 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.079406 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f30c289-8fda-4a5a-a786-68897460562e-podres\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.086994 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.086964 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcq2g\" (UniqueName: \"kubernetes.io/projected/6f30c289-8fda-4a5a-a786-68897460562e-kube-api-access-qcq2g\") pod \"perf-node-gather-daemonset-8n6mr\" (UID: \"6f30c289-8fda-4a5a-a786-68897460562e\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.134979 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.134949 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:20.240589 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.240562 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-5p94q_0223775e-2dd4-4ab0-b979-49b697a046a4/volume-data-source-validator/0.log" Sep 17 02:19:20.407078 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:20.407051 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr"] Sep 17 02:19:20.408495 ip-10-0-143-174 kubenswrapper[2586]: W0917 02:19:20.408472 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f30c289_8fda_4a5a_a786_68897460562e.slice/crio-f24c2abab21bed988c645c7e36f719099b7c7bf1183c0f1b119c8c6b1e654a4b WatchSource:0}: Error finding container f24c2abab21bed988c645c7e36f719099b7c7bf1183c0f1b119c8c6b1e654a4b: Status 404 returned error can't find the container with id f24c2abab21bed988c645c7e36f719099b7c7bf1183c0f1b119c8c6b1e654a4b Sep 17 02:19:21.137304 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:21.137261 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" event={"ID":"6f30c289-8fda-4a5a-a786-68897460562e","Type":"ContainerStarted","Data":"a55598c4bfa910463e47540314ab4b7fe68afc8dfe606b69695de4f4d2428645"} Sep 17 02:19:21.137725 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:21.137436 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:21.137725 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:21.137486 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" event={"ID":"6f30c289-8fda-4a5a-a786-68897460562e","Type":"ContainerStarted","Data":"f24c2abab21bed988c645c7e36f719099b7c7bf1183c0f1b119c8c6b1e654a4b"} Sep 17 02:19:21.154555 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:21.154503 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" podStartSLOduration=2.154488696 podStartE2EDuration="2.154488696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 02:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:19:21.151129906 +0000 UTC m=+2524.907897162" watchObservedRunningTime="2026-09-17 02:19:21.154488696 +0000 UTC m=+2524.911255940" Sep 17 02:19:21.314147 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:21.314117 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pb4gq_ac42c232-24c7-4609-a169-07fe9bb4c2e6/dns/0.log" Sep 17 02:19:21.331898 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:21.331875 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pb4gq_ac42c232-24c7-4609-a169-07fe9bb4c2e6/kube-rbac-proxy/0.log" Sep 17 02:19:21.412878 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:21.412800 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k5mbc_4c2f9cad-e927-484e-9599-a9a84ade862e/dns-node-resolver/0.log" Sep 17 02:19:21.976409 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:21.976380 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rf6rr_d9b4c988-c58d-4e30-94e9-a892d64598b4/node-ca/0.log" Sep 17 02:19:22.931728 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:22.931695 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc_88a99ba0-a5cc-4003-ac5c-be586c05681d/istio-proxy/0.log" Sep 17 02:19:23.340625 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:23.340521 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d8gtj_4a36f446-3677-49e8-b0a6-e0c1eccfe23d/istio-proxy/0.log" Sep 17 02:19:23.343241 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:23.343219 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d8gtj_4a36f446-3677-49e8-b0a6-e0c1eccfe23d/istio-proxy/1.log" Sep 17 02:19:23.364050 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:23.364028 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-ktpnq_5f74e232-dbcf-4586-afbe-eba6a43eea00/payload-pre-processing/0.log" Sep 17 02:19:23.386670 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:23.386648 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-gt5rn_96d36f99-014a-474f-b1e0-74132bb0f4ac/payload-processing/0.log" Sep 17 02:19:23.414083 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:23.414065 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c876f99bb-r6ww6_640963f2-fa1e-44ca-9aa4-e508696d9bf7/router/0.log" Sep 17 02:19:24.215968 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:24.215938 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2djzb_3ce012e5-f6f2-4313-9085-430520d3d06d/serve-healthcheck-canary/0.log" Sep 17 02:19:24.846574 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:24.846541 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-dcgdm_2a42508a-cdab-4526-8b1a-de7088e24b00/insights-operator/0.log" Sep 17 02:19:24.847157 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:24.847137 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-dcgdm_2a42508a-cdab-4526-8b1a-de7088e24b00/insights-operator/1.log" Sep 17 02:19:24.991205 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:24.991176 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kz697_fa85611e-a030-4660-aeb8-5f11fbbdcfc0/kube-rbac-proxy/0.log" Sep 17 02:19:25.008710 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:25.008688 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kz697_fa85611e-a030-4660-aeb8-5f11fbbdcfc0/exporter/0.log" Sep 17 02:19:25.026452 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:25.026421 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kz697_fa85611e-a030-4660-aeb8-5f11fbbdcfc0/extractor/0.log" Sep 17 02:19:27.153761 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:27.153736 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-8n6mr" Sep 17 02:19:27.208600 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:27.208552 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xdl2d_5147556f-49a9-46a5-80bb-49c1009cd777/manager/0.log" Sep 17 02:19:27.432459 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:27.432379 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dqr6g_10322d15-0304-4495-88a9-93e82970c720/manager/0.log" Sep 17 02:19:27.497785 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:27.497760 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7c5n9_c89a9803-f008-4a91-820a-926e14f3bfcb/server/0.log" Sep 17 02:19:27.584652 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:27.584619 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fmtf9_8eb3212a-a53c-45ed-a135-b66c4111693a/manager/0.log" Sep 17 02:19:27.694910 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:27.694842 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5f459ff6-6w7pk_50f81de2-ea37-4ef5-997b-c4120aeb7a0f/manager/0.log" Sep 17 02:19:27.714112 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:27.714091 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5f459ff6-6w7pk_50f81de2-ea37-4ef5-997b-c4120aeb7a0f/copy-manifests/0.log" Sep 17 02:19:29.148524 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:29.148492 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c846775dd-x62nj_89eb875f-5994-4bd5-a74c-ed6fa36b06d9/manager/0.log" Sep 17 02:19:29.552952 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:29.552877 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc_88a99ba0-a5cc-4003-ac5c-be586c05681d/istio-proxy/0.log" Sep 17 02:19:29.953494 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:29.953456 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d8gtj_4a36f446-3677-49e8-b0a6-e0c1eccfe23d/istio-proxy/0.log" Sep 17 02:19:29.955631 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:29.955572 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d8gtj_4a36f446-3677-49e8-b0a6-e0c1eccfe23d/istio-proxy/1.log" Sep 17 02:19:29.966353 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:29.966336 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-ktpnq_5f74e232-dbcf-4586-afbe-eba6a43eea00/payload-pre-processing/0.log" Sep 17 02:19:29.979291 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:29.979272 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-gt5rn_96d36f99-014a-474f-b1e0-74132bb0f4ac/payload-processing/0.log" Sep 17 02:19:29.997317 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:29.997296 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c876f99bb-r6ww6_640963f2-fa1e-44ca-9aa4-e508696d9bf7/router/0.log" Sep 17 02:19:31.430279 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:31.430253 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n7vpc_88a99ba0-a5cc-4003-ac5c-be586c05681d/istio-proxy/0.log" Sep 17 02:19:31.852429 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:31.852398 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d8gtj_4a36f446-3677-49e8-b0a6-e0c1eccfe23d/istio-proxy/0.log" Sep 17 02:19:31.854774 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:31.854753 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d8gtj_4a36f446-3677-49e8-b0a6-e0c1eccfe23d/istio-proxy/1.log" Sep 17 02:19:31.866559 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:31.866539 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-ktpnq_5f74e232-dbcf-4586-afbe-eba6a43eea00/payload-pre-processing/0.log" Sep 17 02:19:31.882284 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:31.882262 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-gt5rn_96d36f99-014a-474f-b1e0-74132bb0f4ac/payload-processing/0.log" Sep 17 02:19:31.899901 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:31.899879 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c876f99bb-r6ww6_640963f2-fa1e-44ca-9aa4-e508696d9bf7/router/0.log" Sep 17 02:19:35.938790 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:35.938762 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxzh6_ba4e55ae-8f06-49da-af0c-18ded6ff0cd5/kube-multus-additional-cni-plugins/0.log" Sep 17 02:19:35.957268 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:35.957246 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxzh6_ba4e55ae-8f06-49da-af0c-18ded6ff0cd5/egress-router-binary-copy/0.log" Sep 17 02:19:35.974736 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:35.974704 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxzh6_ba4e55ae-8f06-49da-af0c-18ded6ff0cd5/cni-plugins/0.log" Sep 17 02:19:35.992872 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:35.992846 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxzh6_ba4e55ae-8f06-49da-af0c-18ded6ff0cd5/bond-cni-plugin/0.log" Sep 17 02:19:36.011808 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:36.011780 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxzh6_ba4e55ae-8f06-49da-af0c-18ded6ff0cd5/routeoverride-cni/0.log" Sep 17 02:19:36.030034 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:36.030014 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxzh6_ba4e55ae-8f06-49da-af0c-18ded6ff0cd5/whereabouts-cni-bincopy/0.log" Sep 17 02:19:36.046957 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:36.046935 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxzh6_ba4e55ae-8f06-49da-af0c-18ded6ff0cd5/whereabouts-cni/0.log" Sep 17 02:19:36.238519 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:36.238443 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xlplg_9c40e778-6932-421e-871c-7654d4d5912f/kube-multus/0.log" Sep 17 02:19:36.261714 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:36.261682 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bjxzr_26801bb3-4b53-4677-a222-d71afd225350/network-metrics-daemon/0.log" Sep 17 02:19:36.281418 ip-10-0-143-174 kubenswrapper[2586]: I0917 02:19:36.281391 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bjxzr_26801bb3-4b53-4677-a222-d71afd225350/kube-rbac-proxy/0.log"