Sep 11 15:06:47.016520 ip-10-0-134-20 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 15:06:47.016533 ip-10-0-134-20 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 15:06:47.016542 ip-10-0-134-20 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 15:06:47.016741 ip-10-0-134-20 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 15:06:57.103239 ip-10-0-134-20 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 15:06:57.103258 ip-10-0-134-20 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 091cbf54ee4546f5b12baf08ad546f4b -- Sep 11 15:08:36.621706 ip-10-0-134-20 systemd[1]: Starting Kubernetes Kubelet... Sep 11 15:08:37.024069 ip-10-0-134-20 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 15:08:37.024069 ip-10-0-134-20 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 15:08:37.024069 ip-10-0-134-20 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 15:08:37.024069 ip-10-0-134-20 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 15:08:37.024069 ip-10-0-134-20 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 15:08:37.026831 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.026759 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 15:08:37.030858 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030837 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:37.030858 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030858 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:37.030858 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030863 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:37.030858 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030866 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030869 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030872 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030875 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030879 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030882 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030885 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030887 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030890 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030893 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030896 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030898 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030901 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030904 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030906 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030909 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030911 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030914 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030917 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030920 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:37.030973 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030922 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030925 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030927 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030930 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030933 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030935 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030939 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030943 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030945 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030948 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030951 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030953 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030956 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030958 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030961 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030964 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030967 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030969 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030972 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:37.031458 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030975 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030978 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030981 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030983 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030986 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030989 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030992 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030994 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030997 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.030999 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031002 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031004 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031007 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031010 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031013 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031015 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031019 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031024 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031027 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:37.031960 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031030 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031033 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031036 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031039 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031042 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031045 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031048 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031050 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031054 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031057 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031060 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031062 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031065 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031067 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031070 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031073 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031075 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031078 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031080 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031083 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:37.032406 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031085 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031088 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031090 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031094 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031096 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031098 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031101 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031104 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031106 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031109 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031111 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031114 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031116 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031119 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031122 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031124 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031127 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031129 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031878 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031886 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:37.032915 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031891 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031894 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031898 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031901 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031903 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031906 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031909 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031912 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031915 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031918 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031921 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031923 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031926 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031928 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031931 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031933 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031938 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031941 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031944 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031947 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:37.033408 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031950 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031952 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031955 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031958 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031961 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031964 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031966 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031969 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031972 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031976 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031980 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031983 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031987 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031991 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031994 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031997 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.031999 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032002 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032005 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:37.033942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032007 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032011 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032013 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032016 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032018 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032021 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032024 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032026 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032029 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032031 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032034 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032037 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032039 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032042 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032044 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032047 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032052 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032056 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032059 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:37.034412 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032062 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032065 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032067 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032070 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032072 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032075 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032078 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032080 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032083 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032085 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032088 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032091 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032093 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032096 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032098 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032101 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032104 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032106 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032109 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032111 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:37.034896 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032114 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032117 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032119 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032122 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032124 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032127 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032129 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032132 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032135 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032138 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032143 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032145 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032148 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032150 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032153 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032156 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032158 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032161 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032163 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032559 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:37.035373 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032566 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032570 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032573 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032575 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032579 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032582 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032584 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032587 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032590 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032593 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032596 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032598 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032600 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032603 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032605 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032608 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032610 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032613 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032615 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032617 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:37.035881 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032620 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032623 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032626 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032629 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032632 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032636 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032639 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032643 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032646 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032648 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032651 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032654 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032657 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032660 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032662 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032665 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032668 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032671 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032674 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:37.036368 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032677 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032680 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032682 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032685 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032687 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032690 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032692 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032695 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032698 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032701 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032704 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032707 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032710 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032712 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032716 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032721 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032725 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032728 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032731 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032733 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:37.036976 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032736 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032738 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032741 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032744 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032747 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032749 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032752 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032754 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032756 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032759 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032763 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032766 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032768 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032771 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032774 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032776 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032779 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032781 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032784 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032786 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:37.037486 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032789 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032791 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032793 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032796 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032799 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032801 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032804 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032807 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032809 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032812 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032815 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032818 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032822 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032825 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032827 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032830 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032832 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032835 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.032837 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032896 2573 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 15:08:37.038019 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032928 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032943 2573 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032949 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032953 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032956 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032960 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032964 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032968 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032971 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032974 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032977 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032980 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032983 2573 flags.go:64] FLAG: --cgroup-root="" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032986 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032989 2573 flags.go:64] FLAG: --client-ca-file="" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032992 2573 flags.go:64] FLAG: --cloud-provider="external" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032995 2573 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.032998 2573 flags.go:64] FLAG: --cluster-domain="" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033001 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033004 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033007 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033010 2573 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033013 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 15:08:37.038535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033016 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033019 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033023 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033025 2573 flags.go:64] FLAG: --contention-profiling="false" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033028 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033031 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033034 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033037 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033040 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033043 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033046 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033050 2573 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033053 2573 flags.go:64] FLAG: --enable-server="true" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033056 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033060 2573 flags.go:64] FLAG: --event-burst="100" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033063 2573 flags.go:64] FLAG: --event-qps="50" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033066 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033069 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033072 2573 flags.go:64] FLAG: --eviction-hard="" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033076 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033078 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033081 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033084 2573 flags.go:64] FLAG: --eviction-soft="" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033087 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033091 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 15:08:37.039071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033094 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033097 2573 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033100 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033102 2573 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033105 2573 flags.go:64] FLAG: --feature-gates="" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033109 2573 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033112 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033114 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033117 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033120 2573 flags.go:64] FLAG: --healthz-port="10248" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033123 2573 flags.go:64] FLAG: --help="false" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033127 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033130 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033132 2573 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033135 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033138 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033142 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033145 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033149 2573 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033152 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033155 2573 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033158 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033161 2573 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033163 2573 flags.go:64] FLAG: --kube-reserved="" Sep 11 15:08:37.039708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033166 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033170 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033173 2573 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033176 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033178 2573 flags.go:64] FLAG: --lock-file="" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033181 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033184 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033188 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033193 2573 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033196 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033198 2573 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033201 2573 flags.go:64] FLAG: --logging-format="text" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033204 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033207 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033210 2573 flags.go:64] FLAG: --manifest-url="" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033213 2573 flags.go:64] FLAG: --manifest-url-header="" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033216 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033219 2573 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033223 2573 flags.go:64] FLAG: --max-pods="110" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033226 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033229 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033232 2573 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033235 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033238 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033241 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 15:08:37.040359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033244 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033251 2573 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033254 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033257 2573 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033260 2573 flags.go:64] FLAG: --pod-cidr="" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033263 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033268 2573 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033271 2573 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033274 2573 flags.go:64] FLAG: --pods-per-core="0" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033281 2573 flags.go:64] FLAG: --port="10250" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033285 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033287 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f477fa47741a08e2" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033291 2573 flags.go:64] FLAG: --qos-reserved="" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033293 2573 flags.go:64] FLAG: --read-only-port="10255" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033296 2573 flags.go:64] FLAG: --register-node="true" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033300 2573 flags.go:64] FLAG: --register-with-taints="" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033304 2573 flags.go:64] FLAG: --registry-burst="10" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033307 2573 flags.go:64] FLAG: --registry-qps="5" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033309 2573 flags.go:64] FLAG: --reserved-cpus="" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033312 2573 flags.go:64] FLAG: --reserved-memory="" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033316 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033319 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033321 2573 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033324 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033327 2573 flags.go:64] FLAG: --runonce="false" Sep 11 15:08:37.040970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033330 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033333 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033336 2573 flags.go:64] FLAG: --seccomp-default="false" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033338 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033341 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033344 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033347 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033350 2573 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033354 2573 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033357 2573 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033359 2573 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033362 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033365 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033368 2573 flags.go:64] FLAG: --system-cgroups="" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033371 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033375 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033378 2573 flags.go:64] FLAG: --tls-cert-file="" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033381 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033385 2573 flags.go:64] FLAG: --tls-min-version="" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033388 2573 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033390 2573 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033393 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033398 2573 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033400 2573 flags.go:64] FLAG: --v="2" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033405 2573 flags.go:64] FLAG: --version="false" Sep 11 15:08:37.041652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033409 2573 flags.go:64] FLAG: --vmodule="" Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033413 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033416 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033546 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033550 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033553 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033556 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033559 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033561 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033564 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033567 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033570 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033572 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033575 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033577 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033582 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033584 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033587 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033590 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:37.042261 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033592 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033594 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033597 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033600 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033602 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033605 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033608 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033610 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033615 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033618 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033623 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033626 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033629 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033632 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033635 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033651 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033655 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033658 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033661 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:37.042744 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033664 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033666 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033669 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033672 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033674 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033677 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033680 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033682 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033685 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033689 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033691 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033694 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033697 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033699 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033702 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033704 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033707 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033709 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033712 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033714 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:37.043217 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033718 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033722 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033725 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033730 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033732 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033736 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033738 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033741 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033744 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033746 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033750 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033754 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033758 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033762 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033765 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033767 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033770 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033772 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033775 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:37.043730 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033777 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033780 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033784 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033786 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033789 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033792 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033795 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033798 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033800 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033803 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033805 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033808 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033811 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033822 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033826 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033829 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033833 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033836 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033838 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033841 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:37.044211 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033844 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033847 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033849 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033852 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.033856 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.033861 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.042468 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.042483 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.042503 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.042511 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042575 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042579 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042583 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042586 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042588 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:37.044715 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042591 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042593 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042596 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042599 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042601 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042604 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042606 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042609 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042612 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042614 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042617 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042627 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042630 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042632 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042635 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042637 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042640 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042643 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042645 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042648 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:37.045090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042650 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042653 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042656 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042659 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042662 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042664 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042667 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042669 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042672 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042675 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042678 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042680 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042683 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042685 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042688 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042691 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042693 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042695 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042698 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042701 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:37.045648 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042703 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042705 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042708 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042713 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042722 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042725 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042727 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042729 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042732 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042735 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042737 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042740 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042742 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042744 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042749 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042754 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042758 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042762 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042765 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:37.046120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042768 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042770 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042773 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042776 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042778 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042781 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042784 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042786 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042789 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042792 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042795 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042798 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042801 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042804 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042807 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042810 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042812 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042814 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042822 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:37.046662 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042825 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042828 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042830 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042833 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042836 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042838 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042840 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042843 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042845 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042848 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042851 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042854 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042857 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042859 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042862 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:37.047142 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.042864 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.042869 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043023 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043029 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043032 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043035 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043038 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043041 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043043 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043046 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043048 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043051 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043054 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043056 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043059 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:37.047560 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043061 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043064 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043073 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043077 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043080 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043083 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043086 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043089 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043091 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043094 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043096 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043100 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043102 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043105 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043108 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043110 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043113 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043115 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043118 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043121 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:37.048006 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043123 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043127 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043130 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043133 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043135 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043138 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043141 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043144 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043147 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043149 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043152 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043155 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043157 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043160 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043163 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043171 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043174 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043177 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043180 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:37.048507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043183 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043185 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043188 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043191 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043194 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043197 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043199 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043202 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043205 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043208 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043210 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043213 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043215 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043218 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043221 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043224 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043226 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043229 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043232 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043235 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:37.048982 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043237 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043240 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043243 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043245 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043248 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043250 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043253 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043255 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043258 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043266 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043269 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043271 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043274 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043276 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043279 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043281 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043284 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043287 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043289 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:37.049493 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043292 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043295 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043299 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043302 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043305 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043307 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043310 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:37.043312 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.043317 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.043417 2573 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.046362 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.047404 2573 server.go:1019] "Starting client certificate rotation" Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.047560 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 15:08:37.050101 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.047609 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 15:08:37.050491 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.050463 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 15:08:37.055874 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.055856 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 15:08:37.066946 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.066923 2573 log.go:25] "Validated CRI v1 runtime API" Sep 11 15:08:37.071230 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.071212 2573 log.go:25] "Validated CRI v1 image API" Sep 11 15:08:37.073011 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.072994 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 15:08:37.079220 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.079201 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9344ae48-2a95-4821-b4f6-7e2a58e74408:/dev/nvme0n1p3 96e4ad9b-9e28-4295-a0be-a0ea1251358a:/dev/nvme0n1p4] Sep 11 15:08:37.079295 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.079218 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 15:08:37.081212 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.081191 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 15:08:37.084605 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.084495 2573 manager.go:216] Machine: {Timestamp:2026-09-11 15:08:37.082961936 +0000 UTC m=+0.364624164 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3076635 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ab08471cf8e4d4622ada26317e5e2 SystemUUID:ec2ab084-71cf-8e4d-4622-ada26317e5e2 BootID:091cbf54-ee45-46f5-b12b-af08ad546f4b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0c:97:12:e2:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0c:97:12:e2:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:f3:d5:e8:fd:ea Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 15:08:37.084605 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.084599 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 15:08:37.084707 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.084676 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 15:08:37.087325 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.087304 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 15:08:37.087473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.087328 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-20.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 15:08:37.088156 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.088146 2573 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 15:08:37.088188 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.088158 2573 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 15:08:37.088188 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.088166 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 15:08:37.089029 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.089019 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 15:08:37.090255 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.090245 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 15:08:37.095069 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.095058 2573 state_mem.go:36] "Initialized new in-memory state store" Sep 11 15:08:37.095175 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.095166 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 15:08:37.095215 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.095206 2573 kubelet.go:486] "Attempting to sync node with API server" Sep 11 15:08:37.095247 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.095235 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 15:08:37.095278 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.095258 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 15:08:37.095278 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.095272 2573 kubelet.go:398] "Adding apiserver pod source" Sep 11 15:08:37.095336 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.095283 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 15:08:37.095795 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.095782 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 15:08:37.095838 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.095805 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 15:08:37.098844 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.098821 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 15:08:37.102801 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.102764 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 15:08:37.102879 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.102830 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 15:08:37.105094 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105074 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 15:08:37.105094 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105092 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 15:08:37.105094 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105099 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 15:08:37.105231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105105 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 15:08:37.105231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105113 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 15:08:37.105231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105118 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 15:08:37.105231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105130 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 15:08:37.105231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105135 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 15:08:37.105231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105141 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 15:08:37.105231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105146 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 15:08:37.105231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105154 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 15:08:37.105231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105159 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 15:08:37.105475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.105329 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rjsjt" Sep 11 15:08:37.106050 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.106039 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 15:08:37.106050 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.106050 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 15:08:37.106896 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.106864 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 15:08:37.106958 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.106939 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-20.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 15:08:37.109390 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.109369 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-20.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 15:08:37.109798 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.109785 2573 server.go:1268] "Started kubelet" Sep 11 15:08:37.109927 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.109878 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 15:08:37.109996 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.109953 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 15:08:37.110044 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.110020 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 15:08:37.110721 ip-10-0-134-20 systemd[1]: Started Kubernetes Kubelet. Sep 11 15:08:37.110945 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.110929 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rjsjt" Sep 11 15:08:37.111148 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.111130 2573 server.go:310] "Adding debug handlers to kubelet server" Sep 11 15:08:37.111752 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.111654 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 15:08:37.114360 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.114339 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 15:08:37.120141 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.120123 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 15:08:37.120228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.120142 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 15:08:37.120315 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.120300 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:37.120892 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.120876 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 15:08:37.120972 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.120896 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 15:08:37.121183 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.120894 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 15:08:37.121183 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.121083 2573 reconstruct.go:97] "Volume reconstruction finished" Sep 11 15:08:37.121183 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.121095 2573 reconciler.go:29] "Reconciler: start to sync state" Sep 11 15:08:37.121383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.121368 2573 factory.go:55] Registering systemd factory Sep 11 15:08:37.121383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.121383 2573 factory.go:223] Registration of the systemd container factory successfully Sep 11 15:08:37.121656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.121642 2573 factory.go:153] Registering CRI-O factory Sep 11 15:08:37.121656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.121658 2573 factory.go:223] Registration of the crio container factory successfully Sep 11 15:08:37.121755 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.121732 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 15:08:37.121788 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.121766 2573 factory.go:103] Registering Raw factory Sep 11 15:08:37.121788 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.121783 2573 manager.go:1196] Started watching for new ooms in manager Sep 11 15:08:37.122412 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.122396 2573 manager.go:349] Starting recovery of all containers Sep 11 15:08:37.124824 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.124802 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:37.129559 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.129539 2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-20.ec2.internal\" not found" node="ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.134566 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.134546 2573 manager.go:354] Recovery completed Sep 11 15:08:37.139841 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.139827 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:37.141510 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.141489 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:37.141572 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.141527 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:37.141572 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.141541 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:37.142120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.142106 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 15:08:37.142120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.142119 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:37.142254 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.142120 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 15:08:37.142254 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.142169 2573 state_mem.go:36] "Initialized new in-memory state store" Sep 11 15:08:37.142924 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.142801 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:37.142984 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.142946 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:37.142984 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.142963 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:37.144396 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.144383 2573 policy_none.go:49] "None policy: Start" Sep 11 15:08:37.144472 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.144403 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 15:08:37.144472 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.144416 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 15:08:37.146454 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.146426 2573 policy_none.go:47] "Start" Sep 11 15:08:37.199904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.190974 2573 manager.go:337] "Starting Device Plugin manager" Sep 11 15:08:37.199904 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.191003 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 15:08:37.199904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.191012 2573 server.go:87] "Starting device plugin registration server" Sep 11 15:08:37.199904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.191280 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 15:08:37.199904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.191294 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 15:08:37.199904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.191373 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 15:08:37.199904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.191475 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 15:08:37.199904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.191485 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 15:08:37.199904 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.191982 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 15:08:37.199904 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.192033 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:37.248715 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.248692 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 15:08:37.249906 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.249889 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 15:08:37.249966 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.249946 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 15:08:37.250012 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.249969 2573 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 15:08:37.250062 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.250044 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 15:08:37.252212 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.252193 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:37.292245 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.292198 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:37.294012 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.293999 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:37.294066 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.294019 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:37.294066 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.294038 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:37.294066 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.294058 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.302455 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.302428 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.302506 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.302463 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-20.ec2.internal\": node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:37.319807 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.319788 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:37.351042 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.351016 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal"] Sep 11 15:08:37.351100 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.351084 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:37.352324 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.352308 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:37.352388 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.352335 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:37.352388 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.352352 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:37.354735 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.354722 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:37.354867 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.354856 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.354903 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.354888 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:37.355390 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.355370 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:37.355390 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.355383 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:37.355520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.355397 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:37.355520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.355407 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:37.355520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.355411 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:37.355520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.355418 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:37.357665 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.357652 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.357734 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.357675 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:37.358352 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.358330 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:37.358352 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.358354 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:37.358507 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.358363 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:37.376948 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.376922 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-20.ec2.internal\" not found" node="ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.380933 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.380917 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-20.ec2.internal\" not found" node="ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.420735 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.420718 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:37.422751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.422736 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dbcd3be065f634dd293745cbb66f2c7e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-20.ec2.internal\" (UID: \"dbcd3be065f634dd293745cbb66f2c7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.422800 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.422759 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5de89eaaf0eccd3705ba8de25ca06353-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"5de89eaaf0eccd3705ba8de25ca06353\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.422800 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.422777 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5de89eaaf0eccd3705ba8de25ca06353-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"5de89eaaf0eccd3705ba8de25ca06353\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.520840 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.520821 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:37.523041 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.523028 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dbcd3be065f634dd293745cbb66f2c7e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-20.ec2.internal\" (UID: \"dbcd3be065f634dd293745cbb66f2c7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.523086 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.523049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5de89eaaf0eccd3705ba8de25ca06353-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"5de89eaaf0eccd3705ba8de25ca06353\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.523086 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.523078 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5de89eaaf0eccd3705ba8de25ca06353-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"5de89eaaf0eccd3705ba8de25ca06353\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.523150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.523115 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5de89eaaf0eccd3705ba8de25ca06353-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"5de89eaaf0eccd3705ba8de25ca06353\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.523150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.523122 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dbcd3be065f634dd293745cbb66f2c7e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-20.ec2.internal\" (UID: \"dbcd3be065f634dd293745cbb66f2c7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.523210 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.523165 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5de89eaaf0eccd3705ba8de25ca06353-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"5de89eaaf0eccd3705ba8de25ca06353\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.621504 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.621463 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:37.679770 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.679749 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.683284 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:37.683258 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" Sep 11 15:08:37.722266 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.722246 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:37.822743 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.822722 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:37.923230 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:37.923189 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:38.023947 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.023929 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found" Sep 11 15:08:38.038897 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.038882 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:38.046805 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.046789 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 15:08:38.046910 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.046890 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 15:08:38.046947 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.046920 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 15:08:38.046947 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.046936 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 15:08:38.096117 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.096098 2573 apiserver.go:52] "Watching apiserver" Sep 11 15:08:38.110503 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.110483 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 15:08:38.111321 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.111303 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-zpl44","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt","openshift-cluster-node-tuning-operator/tuned-sggcw","openshift-dns/node-resolver-tjhlf","openshift-multus/multus-additional-cni-plugins-swbrq","openshift-multus/multus-txqqr","openshift-network-operator/iptables-alerter-7w92z","openshift-image-registry/node-ca-r4zv8","openshift-multus/network-metrics-daemon-m7drl","openshift-network-diagnostics/network-check-target-7z2zb","openshift-ovn-kubernetes/ovnkube-node-mlrgc","kube-system/global-pull-secret-syncer-c6p9x"] Sep 11 15:08:38.113778 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.113763 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:38.115877 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.115860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.115970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.115947 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 15:08:38.116028 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.115947 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 15:08:38.116083 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.116031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qhkst\"" Sep 11 15:08:38.118075 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.117798 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2xlmh\"" Sep 11 15:08:38.118075 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.117926 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 15:08:38.118217 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.118133 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 15:03:38 +0000 UTC" deadline="2028-04-07 16:31:18.749959188 +0000 UTC" Sep 11 15:08:38.118217 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.118193 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 15:08:38.118305 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.118216 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 15:08:38.119793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.118195 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13777h22m40.631767979s" Sep 11 15:08:38.119793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.118526 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.120273 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.120256 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 15:08:38.120385 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.120340 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" Sep 11 15:08:38.120860 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.120838 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:08:38.120979 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.120845 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5mjvx\"" Sep 11 15:08:38.120979 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.120951 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 15:08:38.121102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.121081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.123326 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.123308 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 15:08:38.123326 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.123319 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 15:08:38.123493 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.123308 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.123493 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.123339 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rj2ft\"" Sep 11 15:08:38.125492 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.125466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.125682 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.125665 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 15:08:38.125787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.125674 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 15:08:38.125787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.125778 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 15:08:38.125919 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.125806 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 15:08:38.125919 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.125807 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 15:08:38.125919 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.125866 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pjgqz\"" Sep 11 15:08:38.126845 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.126824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1813d8d4-743c-4b24-b861-aa702a8f5e5b-hosts-file\") pod \"node-resolver-tjhlf\" (UID: \"1813d8d4-743c-4b24-b861-aa702a8f5e5b\") " pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.126912 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.126855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-sysctl-d\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.126912 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.126871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-run\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.126912 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.126894 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-tuned\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.127042 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.126917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1be2d9af-96fa-402c-b577-e51409bf5fe0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.127042 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.126947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14dd5751-9801-4d2e-939c-0930a1293a25-agent-certs\") pod \"konnectivity-agent-zpl44\" (UID: \"14dd5751-9801-4d2e-939c-0930a1293a25\") " pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:38.127042 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.126994 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14dd5751-9801-4d2e-939c-0930a1293a25-konnectivity-ca\") pod \"konnectivity-agent-zpl44\" (UID: \"14dd5751-9801-4d2e-939c-0930a1293a25\") " pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:38.127042 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-socket-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.127198 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-registration-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.127198 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-device-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.127198 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1be2d9af-96fa-402c-b577-e51409bf5fe0-cni-binary-copy\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.127198 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.127198 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-kubernetes\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.127198 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127192 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95hk\" (UniqueName: \"kubernetes.io/projected/b6f0a541-f429-45a1-978f-a4d7ea7dee39-kube-api-access-k95hk\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-sysctl-conf\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-lib-modules\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127235 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1813d8d4-743c-4b24-b861-aa702a8f5e5b-tmp-dir\") pod \"node-resolver-tjhlf\" (UID: \"1813d8d4-743c-4b24-b861-aa702a8f5e5b\") " pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-systemd\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-host\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6f0a541-f429-45a1-978f-a4d7ea7dee39-tmp\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1be2d9af-96fa-402c-b577-e51409bf5fe0-system-cni-dir\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-modprobe-d\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1be2d9af-96fa-402c-b577-e51409bf5fe0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfmnq\" (UniqueName: \"kubernetes.io/projected/1be2d9af-96fa-402c-b577-e51409bf5fe0-kube-api-access-qfmnq\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.127467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4smw\" (UniqueName: \"kubernetes.io/projected/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-kube-api-access-v4smw\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.128115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127489 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkkp\" (UniqueName: \"kubernetes.io/projected/1813d8d4-743c-4b24-b861-aa702a8f5e5b-kube-api-access-mxkkp\") pod \"node-resolver-tjhlf\" (UID: \"1813d8d4-743c-4b24-b861-aa702a8f5e5b\") " pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.128115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1be2d9af-96fa-402c-b577-e51409bf5fe0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.128115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-sys-fs\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.128115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127567 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-sysconfig\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.128115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-sys\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.128115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-var-lib-kubelet\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.128115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127631 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1be2d9af-96fa-402c-b577-e51409bf5fe0-cnibin\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.128115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.128115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.127996 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 15:08:38.128596 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.128247 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v64wk\"" Sep 11 15:08:38.129719 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.129703 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 15:08:38.129719 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.129708 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:08:38.129890 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.129784 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m78gj\"" Sep 11 15:08:38.129962 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.129950 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.130034 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.130006 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 15:08:38.131587 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.131258 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 15:08:38.131587 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.131333 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" Sep 11 15:08:38.131816 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.131797 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xpmng\"" Sep 11 15:08:38.131921 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.131890 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 15:08:38.132016 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.132001 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 15:08:38.132159 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.132144 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 15:08:38.132214 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.132195 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:38.132275 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.132254 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:38.132671 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.132655 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 15:08:38.134601 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.134587 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:38.134654 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.134632 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:38.136950 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.136936 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.138987 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.138971 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 15:08:38.138987 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.138979 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 15:08:38.139102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.139071 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"] Sep 11 15:08:38.139161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.139070 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r6bcz\"" Sep 11 15:08:38.139161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.139145 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:38.139237 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.139223 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:38.139279 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.139148 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 15:08:38.139415 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.139403 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 15:08:38.139644 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.139630 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 15:08:38.139738 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.139649 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal"] Sep 11 15:08:38.139738 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.139670 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 15:08:38.139738 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.139721 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 15:08:38.152489 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.152473 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v5z8v" Sep 11 15:08:38.161342 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.161326 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v5z8v" Sep 11 15:08:38.222011 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.221995 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 15:08:38.227763 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.227745 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-ovn-node-metrics-cert\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.227831 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.227775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6f0a541-f429-45a1-978f-a4d7ea7dee39-tmp\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.227831 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.227801 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:38.227929 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.227843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-etc-openvswitch\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.227929 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.227905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.228023 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.227939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-modprobe-d\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.228074 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.228074 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1be2d9af-96fa-402c-b577-e51409bf5fe0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.228168 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfmnq\" (UniqueName: \"kubernetes.io/projected/1be2d9af-96fa-402c-b577-e51409bf5fe0-kube-api-access-qfmnq\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.228168 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228111 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-log-socket\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.228168 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228112 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 15:08:38.228168 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228135 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.228168 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-ovnkube-script-lib\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.228406 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4smw\" (UniqueName: \"kubernetes.io/projected/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-kube-api-access-v4smw\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.228406 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-modprobe-d\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.228406 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkkp\" (UniqueName: \"kubernetes.io/projected/1813d8d4-743c-4b24-b861-aa702a8f5e5b-kube-api-access-mxkkp\") pod \"node-resolver-tjhlf\" (UID: \"1813d8d4-743c-4b24-b861-aa702a8f5e5b\") " pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.228406 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228243 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1be2d9af-96fa-402c-b577-e51409bf5fe0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.228406 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-var-lib-openvswitch\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.228406 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228328 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-run-openvswitch\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.228406 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228354 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-ovnkube-config\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.228406 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b8d3ca-5688-4b29-a028-5c2cab4a6d74-host-slash\") pod \"iptables-alerter-7w92z\" (UID: \"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74\") " pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-sys-fs\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-sysconfig\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228501 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-sys\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228523 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-daemon-config\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228557 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-kubelet\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fwv7\" (UniqueName: \"kubernetes.io/projected/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-kube-api-access-6fwv7\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228611 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-sysconfig\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228614 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-sys\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228609 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1be2d9af-96fa-402c-b577-e51409bf5fe0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228618 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bcac49b-9179-411c-9dd7-1de6585a7a0f-host\") pod \"node-ca-r4zv8\" (UID: \"3bcac49b-9179-411c-9dd7-1de6585a7a0f\") " pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228696 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1be2d9af-96fa-402c-b577-e51409bf5fe0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228713 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-sys-fs\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-sysctl-d\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-var-lib-kubelet\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228790 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bcac49b-9179-411c-9dd7-1de6585a7a0f-serviceca\") pod \"node-ca-r4zv8\" (UID: \"3bcac49b-9179-411c-9dd7-1de6585a7a0f\") " pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.228797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14dd5751-9801-4d2e-939c-0930a1293a25-agent-certs\") pod \"konnectivity-agent-zpl44\" (UID: \"14dd5751-9801-4d2e-939c-0930a1293a25\") " pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-socket-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228846 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-sysctl-d\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228915 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-run-k8s-cni-cncf-io\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-var-lib-cni-bin\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.228998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-hostroot\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229010 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-socket-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95hk\" (UniqueName: \"kubernetes.io/projected/b6f0a541-f429-45a1-978f-a4d7ea7dee39-kube-api-access-k95hk\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229069 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh8vp\" (UniqueName: \"kubernetes.io/projected/d9b8d3ca-5688-4b29-a028-5c2cab4a6d74-kube-api-access-hh8vp\") pod \"iptables-alerter-7w92z\" (UID: \"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74\") " pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229086 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjl5\" (UniqueName: \"kubernetes.io/projected/9d96e306-761e-4985-b93d-2d8fab445fe1-kube-api-access-nfjl5\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229067 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-cni-bin\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-env-overrides\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229160 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kd67\" (UniqueName: \"kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67\") pod \"network-check-target-7z2zb\" (UID: \"a1646ede-0a03-43a1-8f28-873a9ac43905\") " pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-sysctl-conf\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229200 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-lib-modules\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.229651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-cni-dir\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229316 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-sysctl-conf\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-lib-modules\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1813d8d4-743c-4b24-b861-aa702a8f5e5b-tmp-dir\") pod \"node-resolver-tjhlf\" (UID: \"1813d8d4-743c-4b24-b861-aa702a8f5e5b\") " pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229422 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-systemd\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-host\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-systemd\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1be2d9af-96fa-402c-b577-e51409bf5fe0-system-cni-dir\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-var-lib-cni-multus\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-host\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229551 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-run-netns\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229559 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1be2d9af-96fa-402c-b577-e51409bf5fe0-system-cni-dir\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-run-systemd\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229625 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-cni-netd\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-system-cni-dir\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229651 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1813d8d4-743c-4b24-b861-aa702a8f5e5b-tmp-dir\") pod \"node-resolver-tjhlf\" (UID: \"1813d8d4-743c-4b24-b861-aa702a8f5e5b\") " pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-systemd-units\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.230477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzzsr\" (UniqueName: \"kubernetes.io/projected/3bcac49b-9179-411c-9dd7-1de6585a7a0f-kube-api-access-dzzsr\") pod \"node-ca-r4zv8\" (UID: \"3bcac49b-9179-411c-9dd7-1de6585a7a0f\") " pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229709 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f504ee5-37b1-4ba9-8331-92ce973dc316-kubelet-config\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-run-multus-certs\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229736 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-slash\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f504ee5-37b1-4ba9-8331-92ce973dc316-dbus\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7chk\" (UniqueName: \"kubernetes.io/projected/7fbda896-7786-4cbc-8db2-bfc4dba1537e-kube-api-access-b7chk\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229836 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-var-lib-kubelet\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1be2d9af-96fa-402c-b577-e51409bf5fe0-cnibin\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229904 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-var-lib-kubelet\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229907 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-cnibin\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229927 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1be2d9af-96fa-402c-b577-e51409bf5fe0-cnibin\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-socket-dir-parent\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.229980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-conf-dir\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-node-log\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1813d8d4-743c-4b24-b861-aa702a8f5e5b-hosts-file\") pod \"node-resolver-tjhlf\" (UID: \"1813d8d4-743c-4b24-b861-aa702a8f5e5b\") " pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-run\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.231243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-tuned\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230082 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1813d8d4-743c-4b24-b861-aa702a8f5e5b-hosts-file\") pod \"node-resolver-tjhlf\" (UID: \"1813d8d4-743c-4b24-b861-aa702a8f5e5b\") " pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-run\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1be2d9af-96fa-402c-b577-e51409bf5fe0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-run-ovn\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14dd5751-9801-4d2e-939c-0930a1293a25-konnectivity-ca\") pod \"konnectivity-agent-zpl44\" (UID: \"14dd5751-9801-4d2e-939c-0930a1293a25\") " pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-registration-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-device-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1be2d9af-96fa-402c-b577-e51409bf5fe0-cni-binary-copy\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fbda896-7786-4cbc-8db2-bfc4dba1537e-cni-binary-copy\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230261 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-device-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230272 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-run-netns\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230279 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-registration-dir\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230327 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-kubernetes\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-etc-kubernetes\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230497 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-kubernetes\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9b8d3ca-5688-4b29-a028-5c2cab4a6d74-iptables-alerter-script\") pod \"iptables-alerter-7w92z\" (UID: \"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74\") " pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.231739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230703 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14dd5751-9801-4d2e-939c-0930a1293a25-konnectivity-ca\") pod \"konnectivity-agent-zpl44\" (UID: \"14dd5751-9801-4d2e-939c-0930a1293a25\") " pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:38.232228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230725 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1be2d9af-96fa-402c-b577-e51409bf5fe0-cni-binary-copy\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.232228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.230780 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1be2d9af-96fa-402c-b577-e51409bf5fe0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.232228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.231274 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6f0a541-f429-45a1-978f-a4d7ea7dee39-tmp\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.232228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.231600 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14dd5751-9801-4d2e-939c-0930a1293a25-agent-certs\") pod \"konnectivity-agent-zpl44\" (UID: \"14dd5751-9801-4d2e-939c-0930a1293a25\") " pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:38.232228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.232177 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6f0a541-f429-45a1-978f-a4d7ea7dee39-etc-tuned\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.237063 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.237046 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkkp\" (UniqueName: \"kubernetes.io/projected/1813d8d4-743c-4b24-b861-aa702a8f5e5b-kube-api-access-mxkkp\") pod \"node-resolver-tjhlf\" (UID: \"1813d8d4-743c-4b24-b861-aa702a8f5e5b\") " pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.237311 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.237292 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfmnq\" (UniqueName: \"kubernetes.io/projected/1be2d9af-96fa-402c-b577-e51409bf5fe0-kube-api-access-qfmnq\") pod \"multus-additional-cni-plugins-swbrq\" (UID: \"1be2d9af-96fa-402c-b577-e51409bf5fe0\") " pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.239175 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.239149 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95hk\" (UniqueName: \"kubernetes.io/projected/b6f0a541-f429-45a1-978f-a4d7ea7dee39-kube-api-access-k95hk\") pod \"tuned-sggcw\" (UID: \"b6f0a541-f429-45a1-978f-a4d7ea7dee39\") " pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.239241 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.239185 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4smw\" (UniqueName: \"kubernetes.io/projected/7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241-kube-api-access-v4smw\") pod \"aws-ebs-csi-driver-node-6dnjt\" (UID: \"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.331677 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-run-ovn\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.331836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fbda896-7786-4cbc-8db2-bfc4dba1537e-cni-binary-copy\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.331836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-run-netns\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.331836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-etc-kubernetes\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.331836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331755 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-run-ovn\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.331836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9b8d3ca-5688-4b29-a028-5c2cab4a6d74-iptables-alerter-script\") pod \"iptables-alerter-7w92z\" (UID: \"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74\") " pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.331836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-ovn-node-metrics-cert\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.331836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331824 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-etc-kubernetes\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.331836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331827 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-run-netns\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.331836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331862 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-etc-openvswitch\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-log-socket\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-ovnkube-script-lib\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-var-lib-openvswitch\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331994 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-etc-openvswitch\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-run-openvswitch\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332062 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-run-openvswitch\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332065 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-var-lib-openvswitch\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-ovnkube-config\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.331998 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-log-socket\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b8d3ca-5688-4b29-a028-5c2cab4a6d74-host-slash\") pod \"iptables-alerter-7w92z\" (UID: \"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74\") " pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332417 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-daemon-config\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332492 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-kubelet\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fwv7\" (UniqueName: \"kubernetes.io/projected/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-kube-api-access-6fwv7\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bcac49b-9179-411c-9dd7-1de6585a7a0f-host\") pod \"node-ca-r4zv8\" (UID: \"3bcac49b-9179-411c-9dd7-1de6585a7a0f\") " pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-var-lib-kubelet\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bcac49b-9179-411c-9dd7-1de6585a7a0f-serviceca\") pod \"node-ca-r4zv8\" (UID: \"3bcac49b-9179-411c-9dd7-1de6585a7a0f\") " pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332625 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9b8d3ca-5688-4b29-a028-5c2cab4a6d74-iptables-alerter-script\") pod \"iptables-alerter-7w92z\" (UID: \"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74\") " pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-run-k8s-cni-cncf-io\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-var-lib-cni-bin\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-hostroot\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.332728 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh8vp\" (UniqueName: \"kubernetes.io/projected/d9b8d3ca-5688-4b29-a028-5c2cab4a6d74-kube-api-access-hh8vp\") pod \"iptables-alerter-7w92z\" (UID: \"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74\") " pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.332780 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjl5\" (UniqueName: \"kubernetes.io/projected/9d96e306-761e-4985-b93d-2d8fab445fe1-kube-api-access-nfjl5\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:38.333165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-cni-bin\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.333165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-env-overrides\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.333165 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.332881 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret podName:2f504ee5-37b1-4ba9-8331-92ce973dc316 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:38.832858441 +0000 UTC m=+2.114520674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret") pod "global-pull-secret-syncer-c6p9x" (UID: "2f504ee5-37b1-4ba9-8331-92ce973dc316") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:38.333165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332908 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-ovnkube-script-lib\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.333165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332928 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-ovnkube-config\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.333165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333020 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fbda896-7786-4cbc-8db2-bfc4dba1537e-cni-binary-copy\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.333165 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.333073 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:38.333370 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.333292 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs podName:9d96e306-761e-4985-b93d-2d8fab445fe1 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:38.833273196 +0000 UTC m=+2.114935416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs") pod "network-metrics-daemon-m7drl" (UID: "9d96e306-761e-4985-b93d-2d8fab445fe1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:38.333370 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.332933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kd67\" (UniqueName: \"kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67\") pod \"network-check-target-7z2zb\" (UID: \"a1646ede-0a03-43a1-8f28-873a9ac43905\") " pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:38.333514 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333468 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-env-overrides\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.333584 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333549 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b8d3ca-5688-4b29-a028-5c2cab4a6d74-host-slash\") pod \"iptables-alerter-7w92z\" (UID: \"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74\") " pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.333584 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333568 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-run-k8s-cni-cncf-io\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.333584 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-cni-dir\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.333722 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333611 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-var-lib-cni-multus\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.333722 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333639 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-kubelet\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.333722 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-run-netns\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.333722 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-var-lib-kubelet\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.333722 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333713 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bcac49b-9179-411c-9dd7-1de6585a7a0f-host\") pod \"node-ca-r4zv8\" (UID: \"3bcac49b-9179-411c-9dd7-1de6585a7a0f\") " pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.334009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333732 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-daemon-config\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-run-systemd\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333801 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-run-netns\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333828 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-cni-netd\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333841 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-var-lib-cni-multus\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-system-cni-dir\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333869 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-var-lib-cni-bin\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.333998 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-run-systemd\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334070 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-cni-netd\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334181 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bcac49b-9179-411c-9dd7-1de6585a7a0f-serviceca\") pod \"node-ca-r4zv8\" (UID: \"3bcac49b-9179-411c-9dd7-1de6585a7a0f\") " pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334214 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-systemd-units\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334214 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-cni-bin\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-systemd-units\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334247 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-hostroot\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334278 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzsr\" (UniqueName: \"kubernetes.io/projected/3bcac49b-9179-411c-9dd7-1de6585a7a0f-kube-api-access-dzzsr\") pod \"node-ca-r4zv8\" (UID: \"3bcac49b-9179-411c-9dd7-1de6585a7a0f\") " pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334295 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-system-cni-dir\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334310 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f504ee5-37b1-4ba9-8331-92ce973dc316-kubelet-config\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-cni-dir\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-run-multus-certs\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334397 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-host-run-multus-certs\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-slash\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334468 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f504ee5-37b1-4ba9-8331-92ce973dc316-kubelet-config\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f504ee5-37b1-4ba9-8331-92ce973dc316-dbus\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334501 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-slash\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7chk\" (UniqueName: \"kubernetes.io/projected/7fbda896-7786-4cbc-8db2-bfc4dba1537e-kube-api-access-b7chk\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334565 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334590 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f504ee5-37b1-4ba9-8331-92ce973dc316-dbus\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334615 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-cnibin\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-socket-dir-parent\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334652 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-cnibin\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-conf-dir\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334711 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-node-log\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334719 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-socket-dir-parent\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334766 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-node-log\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.334908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.334794 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fbda896-7786-4cbc-8db2-bfc4dba1537e-multus-conf-dir\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.337332 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.337310 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-ovn-node-metrics-cert\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.339516 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.339500 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:38.339567 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.339525 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:38.339567 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.339538 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6kd67 for pod openshift-network-diagnostics/network-check-target-7z2zb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:38.339648 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.339587 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67 podName:a1646ede-0a03-43a1-8f28-873a9ac43905 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:38.839577158 +0000 UTC m=+2.121239371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6kd67" (UniqueName: "kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67") pod "network-check-target-7z2zb" (UID: "a1646ede-0a03-43a1-8f28-873a9ac43905") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:38.343873 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.343846 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh8vp\" (UniqueName: \"kubernetes.io/projected/d9b8d3ca-5688-4b29-a028-5c2cab4a6d74-kube-api-access-hh8vp\") pod \"iptables-alerter-7w92z\" (UID: \"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74\") " pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.343966 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.343846 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzzsr\" (UniqueName: \"kubernetes.io/projected/3bcac49b-9179-411c-9dd7-1de6585a7a0f-kube-api-access-dzzsr\") pod \"node-ca-r4zv8\" (UID: \"3bcac49b-9179-411c-9dd7-1de6585a7a0f\") " pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.344400 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.344377 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7chk\" (UniqueName: \"kubernetes.io/projected/7fbda896-7786-4cbc-8db2-bfc4dba1537e-kube-api-access-b7chk\") pod \"multus-txqqr\" (UID: \"7fbda896-7786-4cbc-8db2-bfc4dba1537e\") " pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.344400 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.344402 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fwv7\" (UniqueName: \"kubernetes.io/projected/9f2dd3c0-437e-4262-ba18-eb1ce18c35be-kube-api-access-6fwv7\") pod \"ovnkube-node-mlrgc\" (UID: \"9f2dd3c0-437e-4262-ba18-eb1ce18c35be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.344581 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.344377 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjl5\" (UniqueName: \"kubernetes.io/projected/9d96e306-761e-4985-b93d-2d8fab445fe1-kube-api-access-nfjl5\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:38.434818 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.434764 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:38.445767 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.445741 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcd3be065f634dd293745cbb66f2c7e.slice/crio-5921090896dd01a922edfb04326ff05c387e68fc339e0f3bef79286e7bb19289 WatchSource:0}: Error finding container 5921090896dd01a922edfb04326ff05c387e68fc339e0f3bef79286e7bb19289: Status 404 returned error can't find the container with id 5921090896dd01a922edfb04326ff05c387e68fc339e0f3bef79286e7bb19289 Sep 11 15:08:38.448528 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.448502 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dd5751_9801_4d2e_939c_0930a1293a25.slice/crio-c006eac2442317e3f13acdfbe03454342ed331082644de1757f45631c55c17c7 WatchSource:0}: Error finding container c006eac2442317e3f13acdfbe03454342ed331082644de1757f45631c55c17c7: Status 404 returned error can't find the container with id c006eac2442317e3f13acdfbe03454342ed331082644de1757f45631c55c17c7 Sep 11 15:08:38.450022 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.450001 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de89eaaf0eccd3705ba8de25ca06353.slice/crio-d12d988b72fd9575a9f5a0f85ba4a0578048501a16751f36cd0cb8ad25f038b5 WatchSource:0}: Error finding container d12d988b72fd9575a9f5a0f85ba4a0578048501a16751f36cd0cb8ad25f038b5: Status 404 returned error can't find the container with id d12d988b72fd9575a9f5a0f85ba4a0578048501a16751f36cd0cb8ad25f038b5 Sep 11 15:08:38.450375 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.450306 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:08:38.464272 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.464255 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" Sep 11 15:08:38.473096 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.473072 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce0bab8_6d31_4ec7_b8ee_a49a2b81f241.slice/crio-9d818927e9cf17d4b2c0fe998315fe63f7f88ce349fc04e0e1c6c46a533679ce WatchSource:0}: Error finding container 9d818927e9cf17d4b2c0fe998315fe63f7f88ce349fc04e0e1c6c46a533679ce: Status 404 returned error can't find the container with id 9d818927e9cf17d4b2c0fe998315fe63f7f88ce349fc04e0e1c6c46a533679ce Sep 11 15:08:38.481061 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.481045 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sggcw" Sep 11 15:08:38.483165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.483149 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tjhlf" Sep 11 15:08:38.488657 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.488640 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f0a541_f429_45a1_978f_a4d7ea7dee39.slice/crio-a5625418d9d333ec08c76fa7f0e50e3b1c558c9dcb17d1d703b714c1039d7408 WatchSource:0}: Error finding container a5625418d9d333ec08c76fa7f0e50e3b1c558c9dcb17d1d703b714c1039d7408: Status 404 returned error can't find the container with id a5625418d9d333ec08c76fa7f0e50e3b1c558c9dcb17d1d703b714c1039d7408 Sep 11 15:08:38.491591 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.491569 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1813d8d4_743c_4b24_b861_aa702a8f5e5b.slice/crio-dcf004658807611f2130017e7a83faab32b14ea4844e23a66e88e5635136284e WatchSource:0}: Error finding container dcf004658807611f2130017e7a83faab32b14ea4844e23a66e88e5635136284e: Status 404 returned error can't find the container with id dcf004658807611f2130017e7a83faab32b14ea4844e23a66e88e5635136284e Sep 11 15:08:38.498318 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.498302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-swbrq" Sep 11 15:08:38.506591 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.506575 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be2d9af_96fa_402c_b577_e51409bf5fe0.slice/crio-a9daa0627fac2b7d6518be35bb5f259c52df61dcb94e85a755cc03e530b15b78 WatchSource:0}: Error finding container a9daa0627fac2b7d6518be35bb5f259c52df61dcb94e85a755cc03e530b15b78: Status 404 returned error can't find the container with id a9daa0627fac2b7d6518be35bb5f259c52df61dcb94e85a755cc03e530b15b78 Sep 11 15:08:38.514041 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.514027 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-txqqr" Sep 11 15:08:38.521831 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.521805 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbda896_7786_4cbc_8db2_bfc4dba1537e.slice/crio-bf06ffa401702b10758e67ddfe6b21411300a9113a6b9648872b13505f0ffe2f WatchSource:0}: Error finding container bf06ffa401702b10758e67ddfe6b21411300a9113a6b9648872b13505f0ffe2f: Status 404 returned error can't find the container with id bf06ffa401702b10758e67ddfe6b21411300a9113a6b9648872b13505f0ffe2f Sep 11 15:08:38.525718 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.525701 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7w92z" Sep 11 15:08:38.533758 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.533737 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b8d3ca_5688_4b29_a028_5c2cab4a6d74.slice/crio-96f1f77230a9599fa1b4697a031e0b402c29454e4cca5604206a9f153b8dc7ab WatchSource:0}: Error finding container 96f1f77230a9599fa1b4697a031e0b402c29454e4cca5604206a9f153b8dc7ab: Status 404 returned error can't find the container with id 96f1f77230a9599fa1b4697a031e0b402c29454e4cca5604206a9f153b8dc7ab Sep 11 15:08:38.548324 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.548307 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r4zv8" Sep 11 15:08:38.552767 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.552748 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:08:38.557072 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.557053 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcac49b_9179_411c_9dd7_1de6585a7a0f.slice/crio-4a11eb66fc65358456fb7c9d53664d18a59041b051d25d7b90effd2cfe1c58e8 WatchSource:0}: Error finding container 4a11eb66fc65358456fb7c9d53664d18a59041b051d25d7b90effd2cfe1c58e8: Status 404 returned error can't find the container with id 4a11eb66fc65358456fb7c9d53664d18a59041b051d25d7b90effd2cfe1c58e8 Sep 11 15:08:38.560507 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:08:38.560489 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2dd3c0_437e_4262_ba18_eb1ce18c35be.slice/crio-a96984d71e4341461105f944338f5f7cfceee508a2f59fba48fe39738d5ea8f0 WatchSource:0}: Error finding container a96984d71e4341461105f944338f5f7cfceee508a2f59fba48fe39738d5ea8f0: Status 404 returned error can't find the container with id a96984d71e4341461105f944338f5f7cfceee508a2f59fba48fe39738d5ea8f0 Sep 11 15:08:38.562020 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.562002 2573 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-mlrgc_openshift-ovn-kubernetes(9f2dd3c0-437e-4262-ba18-eb1ce18c35be): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 11 15:08:38.563177 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.563163 2573 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-mlrgc_openshift-ovn-kubernetes(9f2dd3c0-437e-4262-ba18-eb1ce18c35be): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 11 15:08:38.566124 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:38.566085 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" podUID="9f2dd3c0-437e-4262-ba18-eb1ce18c35be" Sep 11 15:08:38.628504 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:38.628480 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:40.074838 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.074813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:40.074939 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.074853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:40.074999 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.074946 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:40.074999 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.074978 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:40.074999 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.074995 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs podName:9d96e306-761e-4985-b93d-2d8fab445fe1 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:41.074982306 +0000 UTC m=+3.120307264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs") pod "network-metrics-daemon-m7drl" (UID: "9d96e306-761e-4985-b93d-2d8fab445fe1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:40.075119 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.075024 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret podName:2f504ee5-37b1-4ba9-8331-92ce973dc316 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:41.0750091 +0000 UTC m=+3.120334071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret") pod "global-pull-secret-syncer-c6p9x" (UID: "2f504ee5-37b1-4ba9-8331-92ce973dc316") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:40.175351 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.175320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kd67\" (UniqueName: \"kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67\") pod \"network-check-target-7z2zb\" (UID: \"a1646ede-0a03-43a1-8f28-873a9ac43905\") " pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:40.175499 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.175479 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:40.175499 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.175498 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:40.175598 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.175509 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6kd67 for pod openshift-network-diagnostics/network-check-target-7z2zb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:40.175598 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.175556 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67 podName:a1646ede-0a03-43a1-8f28-873a9ac43905 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:41.175539933 +0000 UTC m=+3.220864904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kd67" (UniqueName: "kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67") pod "network-check-target-7z2zb" (UID: "a1646ede-0a03-43a1-8f28-873a9ac43905") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:40.400450 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.400341 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:03:39 +0000 UTC" deadline="2028-05-15 16:06:47.345319128 +0000 UTC" Sep 11 15:08:40.400450 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.400396 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14688h58m6.944926955s" Sep 11 15:08:40.489007 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.488903 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:40.489141 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.489024 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:40.489413 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.489383 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:40.489610 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.489484 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:40.494486 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.494362 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:40.499399 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.499350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tjhlf" event={"ID":"1813d8d4-743c-4b24-b861-aa702a8f5e5b","Type":"ContainerStarted","Data":"dcf004658807611f2130017e7a83faab32b14ea4844e23a66e88e5635136284e"} Sep 11 15:08:40.501177 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.501126 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sggcw" event={"ID":"b6f0a541-f429-45a1-978f-a4d7ea7dee39","Type":"ContainerStarted","Data":"a5625418d9d333ec08c76fa7f0e50e3b1c558c9dcb17d1d703b714c1039d7408"} Sep 11 15:08:40.515670 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.510314 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" event={"ID":"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241","Type":"ContainerStarted","Data":"9d818927e9cf17d4b2c0fe998315fe63f7f88ce349fc04e0e1c6c46a533679ce"} Sep 11 15:08:40.516561 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.516536 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" event={"ID":"5de89eaaf0eccd3705ba8de25ca06353","Type":"ContainerStarted","Data":"d12d988b72fd9575a9f5a0f85ba4a0578048501a16751f36cd0cb8ad25f038b5"} Sep 11 15:08:40.538796 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.535503 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" event={"ID":"9f2dd3c0-437e-4262-ba18-eb1ce18c35be","Type":"ContainerStarted","Data":"a96984d71e4341461105f944338f5f7cfceee508a2f59fba48fe39738d5ea8f0"} Sep 11 15:08:40.558056 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.557943 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r4zv8" event={"ID":"3bcac49b-9179-411c-9dd7-1de6585a7a0f","Type":"ContainerStarted","Data":"4a11eb66fc65358456fb7c9d53664d18a59041b051d25d7b90effd2cfe1c58e8"} Sep 11 15:08:40.572754 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.567576 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7w92z" event={"ID":"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74","Type":"ContainerStarted","Data":"96f1f77230a9599fa1b4697a031e0b402c29454e4cca5604206a9f153b8dc7ab"} Sep 11 15:08:40.575065 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:40.574453 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" podUID="9f2dd3c0-437e-4262-ba18-eb1ce18c35be" Sep 11 15:08:40.577299 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.577231 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zpl44" event={"ID":"14dd5751-9801-4d2e-939c-0930a1293a25","Type":"ContainerStarted","Data":"c006eac2442317e3f13acdfbe03454342ed331082644de1757f45631c55c17c7"} Sep 11 15:08:40.585041 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.584984 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" event={"ID":"dbcd3be065f634dd293745cbb66f2c7e","Type":"ContainerStarted","Data":"5921090896dd01a922edfb04326ff05c387e68fc339e0f3bef79286e7bb19289"} Sep 11 15:08:40.588207 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.588188 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:40.590207 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.590184 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-txqqr" event={"ID":"7fbda896-7786-4cbc-8db2-bfc4dba1537e","Type":"ContainerStarted","Data":"bf06ffa401702b10758e67ddfe6b21411300a9113a6b9648872b13505f0ffe2f"} Sep 11 15:08:40.600030 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:40.600010 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swbrq" event={"ID":"1be2d9af-96fa-402c-b577-e51409bf5fe0","Type":"ContainerStarted","Data":"a9daa0627fac2b7d6518be35bb5f259c52df61dcb94e85a755cc03e530b15b78"} Sep 11 15:08:41.082766 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:41.082065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:41.082766 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:41.082159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:41.082766 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:41.082273 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:41.082766 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:41.082328 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs podName:9d96e306-761e-4985-b93d-2d8fab445fe1 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:43.082310331 +0000 UTC m=+5.127635308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs") pod "network-metrics-daemon-m7drl" (UID: "9d96e306-761e-4985-b93d-2d8fab445fe1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:41.082766 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:41.082687 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:41.082766 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:41.082735 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret podName:2f504ee5-37b1-4ba9-8331-92ce973dc316 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:43.082720671 +0000 UTC m=+5.128045629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret") pod "global-pull-secret-syncer-c6p9x" (UID: "2f504ee5-37b1-4ba9-8331-92ce973dc316") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:41.182723 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:41.182692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kd67\" (UniqueName: \"kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67\") pod \"network-check-target-7z2zb\" (UID: \"a1646ede-0a03-43a1-8f28-873a9ac43905\") " pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:41.182921 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:41.182860 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:41.182921 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:41.182878 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:41.182921 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:41.182890 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6kd67 for pod openshift-network-diagnostics/network-check-target-7z2zb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:41.183077 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:41.182942 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67 podName:a1646ede-0a03-43a1-8f28-873a9ac43905 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:43.182925743 +0000 UTC m=+5.228250707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kd67" (UniqueName: "kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67") pod "network-check-target-7z2zb" (UID: "a1646ede-0a03-43a1-8f28-873a9ac43905") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:41.210998 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:41.208858 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:41.401185 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:41.401078 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:03:39 +0000 UTC" deadline="2028-03-09 10:08:08.903365452 +0000 UTC" Sep 11 15:08:41.401185 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:41.401134 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13074h59m27.502236005s" Sep 11 15:08:41.486953 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:41.486924 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:41.487135 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:41.487051 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:42.487933 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:42.487071 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:42.487933 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:42.487070 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:42.487933 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:42.487505 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:42.487933 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:42.487653 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:43.099398 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:43.099365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:43.099559 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:43.099418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:43.099559 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:43.099546 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:43.099806 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:43.099561 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:43.099806 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:43.099614 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs podName:9d96e306-761e-4985-b93d-2d8fab445fe1 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:47.099595845 +0000 UTC m=+9.144920821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs") pod "network-metrics-daemon-m7drl" (UID: "9d96e306-761e-4985-b93d-2d8fab445fe1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:43.099806 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:43.099634 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret podName:2f504ee5-37b1-4ba9-8331-92ce973dc316 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:47.099625461 +0000 UTC m=+9.144950419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret") pod "global-pull-secret-syncer-c6p9x" (UID: "2f504ee5-37b1-4ba9-8331-92ce973dc316") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:43.200703 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:43.200672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kd67\" (UniqueName: \"kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67\") pod \"network-check-target-7z2zb\" (UID: \"a1646ede-0a03-43a1-8f28-873a9ac43905\") " pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:43.200867 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:43.200839 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:43.200867 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:43.200858 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:43.200867 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:43.200867 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6kd67 for pod openshift-network-diagnostics/network-check-target-7z2zb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:43.201015 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:43.200916 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67 podName:a1646ede-0a03-43a1-8f28-873a9ac43905 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:47.200903562 +0000 UTC m=+9.246228520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kd67" (UniqueName: "kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67") pod "network-check-target-7z2zb" (UID: "a1646ede-0a03-43a1-8f28-873a9ac43905") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:43.487161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:43.486901 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:43.487161 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:43.487076 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:44.490199 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:44.490172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:44.490700 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:44.490322 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:44.490700 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:44.490364 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:44.490700 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:44.490473 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:45.487230 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:45.487200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:45.487398 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:45.487299 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:46.489539 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:46.489514 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:46.489958 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:46.489620 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:46.489958 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:46.489674 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:46.489958 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:46.489797 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:47.126601 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:47.126090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:47.126601 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:47.126136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:47.126601 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:47.126242 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:47.126601 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:47.126264 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:47.126601 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:47.126302 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret podName:2f504ee5-37b1-4ba9-8331-92ce973dc316 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:55.126288737 +0000 UTC m=+17.171613695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret") pod "global-pull-secret-syncer-c6p9x" (UID: "2f504ee5-37b1-4ba9-8331-92ce973dc316") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:47.126601 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:47.126325 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs podName:9d96e306-761e-4985-b93d-2d8fab445fe1 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:55.126310542 +0000 UTC m=+17.171635501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs") pod "network-metrics-daemon-m7drl" (UID: "9d96e306-761e-4985-b93d-2d8fab445fe1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:47.226960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:47.226929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kd67\" (UniqueName: \"kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67\") pod \"network-check-target-7z2zb\" (UID: \"a1646ede-0a03-43a1-8f28-873a9ac43905\") " pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:47.227153 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:47.227124 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:47.227153 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:47.227153 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:47.227341 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:47.227166 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6kd67 for pod openshift-network-diagnostics/network-check-target-7z2zb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:47.227341 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:47.227226 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67 podName:a1646ede-0a03-43a1-8f28-873a9ac43905 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:55.227209782 +0000 UTC m=+17.272534745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kd67" (UniqueName: "kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67") pod "network-check-target-7z2zb" (UID: "a1646ede-0a03-43a1-8f28-873a9ac43905") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:47.486923 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:47.486851 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:47.487079 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:47.486974 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:48.491159 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:48.491131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:48.491579 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:48.491266 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:48.491579 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:48.491325 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:48.491579 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:48.491419 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:49.487553 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.487404 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:49.487687 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:49.487648 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:49.630887 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.630851 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tjhlf" event={"ID":"1813d8d4-743c-4b24-b861-aa702a8f5e5b","Type":"ContainerStarted","Data":"0cd9a2e043c6b64c82cebff9a4940a29454759173ce8f448afddeec8d84da59b"} Sep 11 15:08:49.632778 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.632745 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sggcw" event={"ID":"b6f0a541-f429-45a1-978f-a4d7ea7dee39","Type":"ContainerStarted","Data":"b3b976ee88ccf81ffdf08560f7de1deb44e2a07152ccd7e7ef2234a90dff0b3b"} Sep 11 15:08:49.634473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.634446 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" event={"ID":"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241","Type":"ContainerStarted","Data":"b7629061dbb08a79e26df850031c713ec61f05886a542deac5505897accb0bf2"} Sep 11 15:08:49.636155 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.636126 2573 generic.go:356] "Generic (PLEG): container finished" podID="5de89eaaf0eccd3705ba8de25ca06353" containerID="422ca760314305ba473e6f3da7daadddb298b357b26ce97baf4486625f81ac12" exitCode=0 Sep 11 15:08:49.636267 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.636214 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" event={"ID":"5de89eaaf0eccd3705ba8de25ca06353","Type":"ContainerDied","Data":"422ca760314305ba473e6f3da7daadddb298b357b26ce97baf4486625f81ac12"} Sep 11 15:08:49.637850 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.637827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r4zv8" event={"ID":"3bcac49b-9179-411c-9dd7-1de6585a7a0f","Type":"ContainerStarted","Data":"0cd1a44bc21c60e8be202b33caebfb38b909c3f49abefaf857fa6c3290727d2a"} Sep 11 15:08:49.639365 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.639344 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zpl44" event={"ID":"14dd5751-9801-4d2e-939c-0930a1293a25","Type":"ContainerStarted","Data":"4b0ea4e2c3593e0eaf572fbfe223b848423c47d7f04414b58900620f7a2d34c4"} Sep 11 15:08:49.640926 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.640902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" event={"ID":"dbcd3be065f634dd293745cbb66f2c7e","Type":"ContainerStarted","Data":"b54eb188d6738eb497d68a4f204ee9cb2e989e4d1df6da6473a6dc73b982578f"} Sep 11 15:08:49.642484 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.642460 2573 generic.go:356] "Generic (PLEG): container finished" podID="1be2d9af-96fa-402c-b577-e51409bf5fe0" containerID="1eca9ec8487c943f83a20669b94337299ce324df3a535952c5feba0bd04b4b5d" exitCode=0 Sep 11 15:08:49.642576 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.642494 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swbrq" event={"ID":"1be2d9af-96fa-402c-b577-e51409bf5fe0","Type":"ContainerDied","Data":"1eca9ec8487c943f83a20669b94337299ce324df3a535952c5feba0bd04b4b5d"} Sep 11 15:08:49.651695 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.651651 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tjhlf" podStartSLOduration=2.419946963 podStartE2EDuration="11.651635103s" totalImagesPullingTime="9.23168814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:49.651335363 +0000 UTC m=+11.696660343" watchObservedRunningTime="2026-09-11 15:08:49.651635103 +0000 UTC m=+11.696960083" Sep 11 15:08:49.666538 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.666489 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" podStartSLOduration=10.666473136 podStartE2EDuration="10.666473136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:49.666439639 +0000 UTC m=+11.711764619" watchObservedRunningTime="2026-09-11 15:08:49.666473136 +0000 UTC m=+11.711798147" Sep 11 15:08:49.701382 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.701329 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r4zv8" podStartSLOduration=2.535675904 podStartE2EDuration="11.701312524s" totalImagesPullingTime="9.16563662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:49.700921173 +0000 UTC m=+11.746246153" watchObservedRunningTime="2026-09-11 15:08:49.701312524 +0000 UTC m=+11.746637505" Sep 11 15:08:49.750742 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:49.750705 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zpl44" podStartSLOduration=2.503837596 podStartE2EDuration="11.750694468s" totalImagesPullingTime="9.246856872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:49.733277082 +0000 UTC m=+11.778602063" watchObservedRunningTime="2026-09-11 15:08:49.750694468 +0000 UTC m=+11.796019482" Sep 11 15:08:50.487138 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:50.487112 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:50.487287 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:50.487129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:50.487363 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:50.487313 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:50.487477 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:50.487450 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:50.646552 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:50.646461 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7w92z" event={"ID":"d9b8d3ca-5688-4b29-a028-5c2cab4a6d74","Type":"ContainerStarted","Data":"05ca54a3201660bdf2b60d05726fdff3cc0b9f78fc87d53a4a36660ca34821bd"} Sep 11 15:08:50.660416 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:50.660358 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sggcw" podStartSLOduration=3.409175184 podStartE2EDuration="12.660337188s" totalImagesPullingTime="9.251162004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:49.750943984 +0000 UTC m=+11.796268964" watchObservedRunningTime="2026-09-11 15:08:50.660337188 +0000 UTC m=+12.705662170" Sep 11 15:08:50.660803 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:50.660754 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7w92z" podStartSLOduration=3.470720376 podStartE2EDuration="12.660747222s" totalImagesPullingTime="9.190026846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:50.660584356 +0000 UTC m=+12.705909346" watchObservedRunningTime="2026-09-11 15:08:50.660747222 +0000 UTC m=+12.706072206" Sep 11 15:08:51.486899 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:51.486863 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:51.487065 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:51.486979 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:51.830393 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:51.830354 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:51.831089 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:51.831070 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:52.487227 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:52.487199 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:52.487392 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:52.487215 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:52.487392 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:52.487321 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:52.487518 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:52.487406 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:53.487031 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:53.487005 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:53.487378 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:53.487101 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:53.654279 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:53.654215 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" event={"ID":"5de89eaaf0eccd3705ba8de25ca06353","Type":"ContainerStarted","Data":"ba8d96d15f0151de20bf30e735b129c342cf4834d21e0b17b92fdec8c6c67121"} Sep 11 15:08:54.429233 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:54.428747 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 15:08:54.434305 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:54.434212 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T15:08:54.42876638Z","UUID":"bff5ac58-1ef6-480d-990d-b42388f1877e","Handler":null,"Name":"","Endpoint":""} Sep 11 15:08:54.436071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:54.436055 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 15:08:54.436170 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:54.436081 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 15:08:54.490972 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:54.490920 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:54.490972 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:54.490937 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:54.491430 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:54.491030 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:54.491430 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:54.491400 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:54.659006 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:54.658969 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" event={"ID":"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241","Type":"ContainerStarted","Data":"41183f5741475f5e8addd710dc0d123d4d9c5603b2c9506e0ed4d0655570f83a"} Sep 11 15:08:54.662632 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:54.662597 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-txqqr" event={"ID":"7fbda896-7786-4cbc-8db2-bfc4dba1537e","Type":"ContainerStarted","Data":"789f3bc9addd8294d25208e7b5ad0666468724646f413c7f3a4787f3a336e21b"} Sep 11 15:08:54.681597 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:54.681536 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" podStartSLOduration=15.681516342 podStartE2EDuration="15.681516342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:53.670906184 +0000 UTC m=+15.716231165" watchObservedRunningTime="2026-09-11 15:08:54.681516342 +0000 UTC m=+16.726841323" Sep 11 15:08:55.191386 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:55.191350 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:55.191533 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:55.191410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:55.191586 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:55.191541 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:55.191622 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:55.191609 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret podName:2f504ee5-37b1-4ba9-8331-92ce973dc316 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:11.191581311 +0000 UTC m=+33.236906272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret") pod "global-pull-secret-syncer-c6p9x" (UID: "2f504ee5-37b1-4ba9-8331-92ce973dc316") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:55.192029 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:55.192009 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:55.192098 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:55.192063 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs podName:9d96e306-761e-4985-b93d-2d8fab445fe1 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:11.192048373 +0000 UTC m=+33.237373336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs") pod "network-metrics-daemon-m7drl" (UID: "9d96e306-761e-4985-b93d-2d8fab445fe1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:55.292423 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:55.292391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kd67\" (UniqueName: \"kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67\") pod \"network-check-target-7z2zb\" (UID: \"a1646ede-0a03-43a1-8f28-873a9ac43905\") " pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:55.292598 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:55.292531 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:55.292598 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:55.292552 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:55.292598 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:55.292565 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6kd67 for pod openshift-network-diagnostics/network-check-target-7z2zb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:55.292753 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:55.292617 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67 podName:a1646ede-0a03-43a1-8f28-873a9ac43905 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:11.29259856 +0000 UTC m=+33.337923522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kd67" (UniqueName: "kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67") pod "network-check-target-7z2zb" (UID: "a1646ede-0a03-43a1-8f28-873a9ac43905") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:55.486569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:55.486502 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:55.486699 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:55.486623 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:56.490590 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:56.490400 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:56.491002 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:56.490685 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:56.491002 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:56.490793 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:56.491002 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:56.490889 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:56.670579 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:56.670548 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" event={"ID":"7ce0bab8-6d31-4ec7-b8ee-a49a2b81f241","Type":"ContainerStarted","Data":"324fb9277a625389b2a6e379a8f3342bd1f3cf599826ed7491181a9ee44fea16"} Sep 11 15:08:56.689808 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:56.689744 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dnjt" podStartSLOduration=2.643499036 podStartE2EDuration="18.689726812s" totalImagesPullingTime="16.046227776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:56.689554748 +0000 UTC m=+18.734879729" watchObservedRunningTime="2026-09-11 15:08:56.689726812 +0000 UTC m=+18.735051793" Sep 11 15:08:56.690140 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:56.690112 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-txqqr" podStartSLOduration=4.790365352 podStartE2EDuration="18.690103221s" totalImagesPullingTime="13.899737869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:54.682305637 +0000 UTC m=+16.727630640" watchObservedRunningTime="2026-09-11 15:08:56.690103221 +0000 UTC m=+18.735428201" Sep 11 15:08:57.486553 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:57.486523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:57.486738 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:57.486643 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:58.490929 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:58.490894 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:08:58.491368 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:58.491323 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:08:58.491425 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:58.491358 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:08:58.491476 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:58.491431 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:08:59.486745 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:59.486714 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:08:59.486929 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:08:59.486860 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:08:59.678971 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:59.678934 2573 generic.go:356] "Generic (PLEG): container finished" podID="1be2d9af-96fa-402c-b577-e51409bf5fe0" containerID="8e13ffc1128d8903e2e5f66db8eca3119e8114e44e56564b4f0dfe021273d5e4" exitCode=0 Sep 11 15:08:59.679523 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:59.678999 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swbrq" event={"ID":"1be2d9af-96fa-402c-b577-e51409bf5fe0","Type":"ContainerDied","Data":"8e13ffc1128d8903e2e5f66db8eca3119e8114e44e56564b4f0dfe021273d5e4"} Sep 11 15:08:59.819740 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:59.819673 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:08:59.819909 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:59.819828 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 15:08:59.820321 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:08:59.820277 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zpl44" Sep 11 15:09:00.489550 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:00.489523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:00.489812 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:00.489523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:00.489812 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:00.489627 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:09:00.489812 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:00.489715 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:09:01.486665 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:01.486636 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:01.487247 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:01.486748 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:09:02.490676 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:02.487644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:02.490676 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:02.487650 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:02.490676 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:02.487932 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:09:02.490676 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:02.488032 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:09:03.487294 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:03.487110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:03.487368 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:03.487350 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:09:03.689369 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:03.689341 2573 generic.go:356] "Generic (PLEG): container finished" podID="1be2d9af-96fa-402c-b577-e51409bf5fe0" containerID="192483bca4ddc6c88a74e40f1ff7c1351ed506cbe13e645813b405ca8032888f" exitCode=0 Sep 11 15:09:03.689965 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:03.689426 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swbrq" event={"ID":"1be2d9af-96fa-402c-b577-e51409bf5fe0","Type":"ContainerDied","Data":"192483bca4ddc6c88a74e40f1ff7c1351ed506cbe13e645813b405ca8032888f"} Sep 11 15:09:03.692685 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:03.692660 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" event={"ID":"9f2dd3c0-437e-4262-ba18-eb1ce18c35be","Type":"ContainerStarted","Data":"d98e4cbca123f1a067aeb32b56019de435583f823dbcbe9c72f2347fde0dc3b8"} Sep 11 15:09:03.692815 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:03.692726 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" event={"ID":"9f2dd3c0-437e-4262-ba18-eb1ce18c35be","Type":"ContainerStarted","Data":"fd6b712a192789a840b3b45c402ebeff282704948252be3d56ab7cfae9d2a488"} Sep 11 15:09:03.692815 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:03.692758 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" event={"ID":"9f2dd3c0-437e-4262-ba18-eb1ce18c35be","Type":"ContainerStarted","Data":"c1695c9e372da505819b3d96b2b150d3ff4e257efa0644f48325fa2f3c6abab2"} Sep 11 15:09:03.692815 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:03.692810 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" event={"ID":"9f2dd3c0-437e-4262-ba18-eb1ce18c35be","Type":"ContainerStarted","Data":"c1218f6c887037d76e3dc4862851f0a29605c1ef338e0a749a204221237416fa"} Sep 11 15:09:03.692917 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:03.692839 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" event={"ID":"9f2dd3c0-437e-4262-ba18-eb1ce18c35be","Type":"ContainerStarted","Data":"66f716bba937115721e5827df1c35484a2246cf3476201ee679497ad68a7007c"} Sep 11 15:09:03.692917 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:03.692867 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" event={"ID":"9f2dd3c0-437e-4262-ba18-eb1ce18c35be","Type":"ContainerStarted","Data":"b557b6946ec9e22441420bf6d6a291dd8349ecc45510623d8f32ed8ff299c8a1"} Sep 11 15:09:04.489506 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:04.489484 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:04.489669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:04.489646 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:04.492241 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:04.490003 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:09:04.492241 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:04.490087 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:09:04.697001 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:04.696952 2573 generic.go:356] "Generic (PLEG): container finished" podID="1be2d9af-96fa-402c-b577-e51409bf5fe0" containerID="0bd5b85c40dfaf73d741af05719f0be3971980b732ec1dec6fbe2d833a62b8e1" exitCode=0 Sep 11 15:09:04.697271 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:04.697002 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swbrq" event={"ID":"1be2d9af-96fa-402c-b577-e51409bf5fe0","Type":"ContainerDied","Data":"0bd5b85c40dfaf73d741af05719f0be3971980b732ec1dec6fbe2d833a62b8e1"} Sep 11 15:09:05.486783 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:05.486741 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:05.486945 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:05.486845 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:09:05.704289 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:05.704253 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" event={"ID":"9f2dd3c0-437e-4262-ba18-eb1ce18c35be","Type":"ContainerStarted","Data":"c38704d8e540fb817cff40d7c79a29f493ff07ba22db146bf2ee1d78470adf0d"} Sep 11 15:09:06.490117 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:06.490092 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:06.490294 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:06.490094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:06.490294 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:06.490218 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:09:06.490391 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:06.490292 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:09:07.486712 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:07.486678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:07.487114 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:07.486819 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:09:08.490910 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:08.490664 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:08.491362 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:08.490889 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:08.491362 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:08.491022 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:09:08.491362 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:08.491079 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:09:08.714928 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:08.714891 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" event={"ID":"9f2dd3c0-437e-4262-ba18-eb1ce18c35be","Type":"ContainerStarted","Data":"87e7bf3f90826f1a913013c949aa00dedfd731ddd6b07d812d112ef99c5c5022"} Sep 11 15:09:08.715197 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:08.715178 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:09:08.715274 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:08.715266 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:09:08.732027 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:08.732004 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:09:08.744859 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:08.744762 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" podStartSLOduration=7.55437699 podStartE2EDuration="30.744748125s" totalImagesPullingTime="23.190371135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-09-11 15:09:08.744533246 +0000 UTC m=+30.789858253" watchObservedRunningTime="2026-09-11 15:09:08.744748125 +0000 UTC m=+30.790073105" Sep 11 15:09:09.487328 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:09.487295 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:09.487522 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:09.487406 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:09:09.718531 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:09.718502 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:09:09.734212 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:09.734177 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:09:09.879838 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:09.879808 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c6p9x"] Sep 11 15:09:09.880011 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:09.879920 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:09.880011 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:09.879998 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:09:09.882546 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:09.882524 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m7drl"] Sep 11 15:09:09.882681 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:09.882646 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:09.882823 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:09.882762 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:09:09.883450 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:09.883277 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7z2zb"] Sep 11 15:09:09.883450 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:09.883358 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:09.883450 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:09.883424 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:09:10.722611 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:10.722546 2573 generic.go:356] "Generic (PLEG): container finished" podID="1be2d9af-96fa-402c-b577-e51409bf5fe0" containerID="74341b110a26b957b0cee229bd8c495ff22847dc6b6fa07d2a706115c2820709" exitCode=0 Sep 11 15:09:10.723095 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:10.722642 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swbrq" event={"ID":"1be2d9af-96fa-402c-b577-e51409bf5fe0","Type":"ContainerDied","Data":"74341b110a26b957b0cee229bd8c495ff22847dc6b6fa07d2a706115c2820709"} Sep 11 15:09:11.209512 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:11.209487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:11.209654 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:11.209518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:11.209654 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.209611 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:09:11.209654 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.209613 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:09:11.209654 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.209653 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret podName:2f504ee5-37b1-4ba9-8331-92ce973dc316 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:43.2096415 +0000 UTC m=+65.254966458 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret") pod "global-pull-secret-syncer-c6p9x" (UID: "2f504ee5-37b1-4ba9-8331-92ce973dc316") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:09:11.209837 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.209667 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs podName:9d96e306-761e-4985-b93d-2d8fab445fe1 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:43.209660599 +0000 UTC m=+65.254985557 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs") pod "network-metrics-daemon-m7drl" (UID: "9d96e306-761e-4985-b93d-2d8fab445fe1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:09:11.310713 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:11.310690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kd67\" (UniqueName: \"kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67\") pod \"network-check-target-7z2zb\" (UID: \"a1646ede-0a03-43a1-8f28-873a9ac43905\") " pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:11.310984 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.310814 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:09:11.310984 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.310829 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:09:11.310984 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.310838 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6kd67 for pod openshift-network-diagnostics/network-check-target-7z2zb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:09:11.310984 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.310878 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67 podName:a1646ede-0a03-43a1-8f28-873a9ac43905 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:43.310865125 +0000 UTC m=+65.356190083 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kd67" (UniqueName: "kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67") pod "network-check-target-7z2zb" (UID: "a1646ede-0a03-43a1-8f28-873a9ac43905") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:09:11.487502 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:11.487433 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:11.487616 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:11.487532 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:11.487616 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:11.487539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:11.487616 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.487532 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:09:11.487801 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.487642 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:09:11.487801 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:11.487726 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:09:11.728521 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:11.728480 2573 generic.go:356] "Generic (PLEG): container finished" podID="1be2d9af-96fa-402c-b577-e51409bf5fe0" containerID="4fca2ff92e2454229ca76df723dac7297316231f481e3e0cd66e22a434822360" exitCode=0 Sep 11 15:09:11.728989 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:11.728555 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swbrq" event={"ID":"1be2d9af-96fa-402c-b577-e51409bf5fe0","Type":"ContainerDied","Data":"4fca2ff92e2454229ca76df723dac7297316231f481e3e0cd66e22a434822360"} Sep 11 15:09:12.734412 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:12.734245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swbrq" event={"ID":"1be2d9af-96fa-402c-b577-e51409bf5fe0","Type":"ContainerStarted","Data":"f2802703f331e0cc900af8111405ebca268d1391f4d5e0a46775f74c457fee38"} Sep 11 15:09:12.755680 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:12.755636 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-swbrq" podStartSLOduration=6.427344205 podStartE2EDuration="34.75562156s" totalImagesPullingTime="28.328277355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:12.755132185 +0000 UTC m=+34.800457164" watchObservedRunningTime="2026-09-11 15:09:12.75562156 +0000 UTC m=+34.800946539" Sep 11 15:09:13.486977 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:13.486951 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:13.486977 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:13.486975 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:13.487164 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:13.486951 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:13.487164 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:13.487052 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7z2zb" podUID="a1646ede-0a03-43a1-8f28-873a9ac43905" Sep 11 15:09:13.487164 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:13.487124 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m7drl" podUID="9d96e306-761e-4985-b93d-2d8fab445fe1" Sep 11 15:09:13.487279 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:13.487178 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6p9x" podUID="2f504ee5-37b1-4ba9-8331-92ce973dc316" Sep 11 15:09:15.239671 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.239650 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeReady" Sep 11 15:09:15.240009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.239755 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 15:09:15.292839 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.292746 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-smjz7"] Sep 11 15:09:15.310463 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.310442 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p94fh"] Sep 11 15:09:15.310606 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.310584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-smjz7" Sep 11 15:09:15.313165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.313147 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 15:09:15.313283 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.313195 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6rfh2\"" Sep 11 15:09:15.313283 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.313236 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 15:09:15.313399 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.313303 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 15:09:15.324738 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.324706 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-smjz7"] Sep 11 15:09:15.324853 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.324746 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p94fh"] Sep 11 15:09:15.324913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.324850 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.327183 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.327165 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zldr7\"" Sep 11 15:09:15.327274 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.327187 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 15:09:15.327274 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.327165 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 15:09:15.441429 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.441399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-config-volume\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.441534 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.441435 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6m4\" (UniqueName: \"kubernetes.io/projected/522b2527-f838-4d7d-95cf-6f2ce8ff4bd4-kube-api-access-5z6m4\") pod \"ingress-canary-smjz7\" (UID: \"522b2527-f838-4d7d-95cf-6f2ce8ff4bd4\") " pod="openshift-ingress-canary/ingress-canary-smjz7" Sep 11 15:09:15.441534 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.441498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.441534 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.441519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-tmp-dir\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.441630 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.441561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wch\" (UniqueName: \"kubernetes.io/projected/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-kube-api-access-q2wch\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.441630 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.441576 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/522b2527-f838-4d7d-95cf-6f2ce8ff4bd4-cert\") pod \"ingress-canary-smjz7\" (UID: \"522b2527-f838-4d7d-95cf-6f2ce8ff4bd4\") " pod="openshift-ingress-canary/ingress-canary-smjz7" Sep 11 15:09:15.486826 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.486800 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:15.486929 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.486804 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:15.487001 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.486807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:15.489515 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.489489 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 15:09:15.489601 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.489528 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 15:09:15.489601 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.489554 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2f6h7\"" Sep 11 15:09:15.489686 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.489661 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4h8rg\"" Sep 11 15:09:15.489725 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.489688 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 15:09:15.489808 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.489792 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 15:09:15.542620 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.542603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wch\" (UniqueName: \"kubernetes.io/projected/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-kube-api-access-q2wch\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.542729 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.542627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/522b2527-f838-4d7d-95cf-6f2ce8ff4bd4-cert\") pod \"ingress-canary-smjz7\" (UID: \"522b2527-f838-4d7d-95cf-6f2ce8ff4bd4\") " pod="openshift-ingress-canary/ingress-canary-smjz7" Sep 11 15:09:15.542729 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.542664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-config-volume\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.542729 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.542681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6m4\" (UniqueName: \"kubernetes.io/projected/522b2527-f838-4d7d-95cf-6f2ce8ff4bd4-kube-api-access-5z6m4\") pod \"ingress-canary-smjz7\" (UID: \"522b2527-f838-4d7d-95cf-6f2ce8ff4bd4\") " pod="openshift-ingress-canary/ingress-canary-smjz7" Sep 11 15:09:15.542729 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.542713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.542990 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.542740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-tmp-dir\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.542990 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:15.542837 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:15.542990 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:15.542908 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls podName:07bd3b2e-2168-40ec-8fcf-6a55bfa7468c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:16.042888903 +0000 UTC m=+38.088213861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls") pod "dns-default-p94fh" (UID: "07bd3b2e-2168-40ec-8fcf-6a55bfa7468c") : secret "dns-default-metrics-tls" not found Sep 11 15:09:15.543140 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.543083 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-tmp-dir\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.543268 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.543251 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-config-volume\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.546892 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.546871 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/522b2527-f838-4d7d-95cf-6f2ce8ff4bd4-cert\") pod \"ingress-canary-smjz7\" (UID: \"522b2527-f838-4d7d-95cf-6f2ce8ff4bd4\") " pod="openshift-ingress-canary/ingress-canary-smjz7" Sep 11 15:09:15.554108 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.554088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wch\" (UniqueName: \"kubernetes.io/projected/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-kube-api-access-q2wch\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:15.554311 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.554296 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6m4\" (UniqueName: \"kubernetes.io/projected/522b2527-f838-4d7d-95cf-6f2ce8ff4bd4-kube-api-access-5z6m4\") pod \"ingress-canary-smjz7\" (UID: \"522b2527-f838-4d7d-95cf-6f2ce8ff4bd4\") " pod="openshift-ingress-canary/ingress-canary-smjz7" Sep 11 15:09:15.620195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.620175 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-smjz7" Sep 11 15:09:15.945827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:15.945798 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-smjz7"] Sep 11 15:09:15.946830 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:15.946800 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522b2527_f838_4d7d_95cf_6f2ce8ff4bd4.slice/crio-40deda1a0155f1747a4fa4f524a9339de5c488ec59657ebfa33db5b72aad53d7 WatchSource:0}: Error finding container 40deda1a0155f1747a4fa4f524a9339de5c488ec59657ebfa33db5b72aad53d7: Status 404 returned error can't find the container with id 40deda1a0155f1747a4fa4f524a9339de5c488ec59657ebfa33db5b72aad53d7 Sep 11 15:09:16.047095 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:16.047073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:16.047187 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:16.047175 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:16.047239 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:16.047228 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls podName:07bd3b2e-2168-40ec-8fcf-6a55bfa7468c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:17.047211374 +0000 UTC m=+39.092536332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls") pod "dns-default-p94fh" (UID: "07bd3b2e-2168-40ec-8fcf-6a55bfa7468c") : secret "dns-default-metrics-tls" not found Sep 11 15:09:16.744833 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:16.744595 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-smjz7" event={"ID":"522b2527-f838-4d7d-95cf-6f2ce8ff4bd4","Type":"ContainerStarted","Data":"40deda1a0155f1747a4fa4f524a9339de5c488ec59657ebfa33db5b72aad53d7"} Sep 11 15:09:17.054064 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:17.053979 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:17.054218 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:17.054124 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:17.054218 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:17.054193 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls podName:07bd3b2e-2168-40ec-8fcf-6a55bfa7468c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:19.05417758 +0000 UTC m=+41.099502559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls") pod "dns-default-p94fh" (UID: "07bd3b2e-2168-40ec-8fcf-6a55bfa7468c") : secret "dns-default-metrics-tls" not found Sep 11 15:09:19.070196 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:19.070110 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:19.070660 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:19.070262 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:19.070660 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:19.070331 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls podName:07bd3b2e-2168-40ec-8fcf-6a55bfa7468c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:23.070316147 +0000 UTC m=+45.115641106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls") pod "dns-default-p94fh" (UID: "07bd3b2e-2168-40ec-8fcf-6a55bfa7468c") : secret "dns-default-metrics-tls" not found Sep 11 15:09:19.751932 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:19.751901 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-smjz7" event={"ID":"522b2527-f838-4d7d-95cf-6f2ce8ff4bd4","Type":"ContainerStarted","Data":"b77ab03073ee7639a25b7074af91ebc5936cbd52a32bc7f2b7fbdc619283ca76"} Sep 11 15:09:19.771438 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:19.771396 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-smjz7" podStartSLOduration=1.94116146 podStartE2EDuration="4.771382791s" totalImagesPullingTime="2.830221331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:19.770090914 +0000 UTC m=+41.815415894" watchObservedRunningTime="2026-09-11 15:09:19.771382791 +0000 UTC m=+41.816707770" Sep 11 15:09:21.582475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.582441 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78"] Sep 11 15:09:21.585071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.585055 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.587325 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.587296 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 15:09:21.587443 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.587385 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 15:09:21.587616 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.587600 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:09:21.587694 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.587629 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 15:09:21.588273 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.588254 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bhxb6\"" Sep 11 15:09:21.597239 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.597221 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78"] Sep 11 15:09:21.686647 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.686620 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a709ba86-240f-4cdf-afc4-69a224583c98-serving-cert\") pod \"service-ca-operator-cf7c776b7-kkr78\" (UID: \"a709ba86-240f-4cdf-afc4-69a224583c98\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.686749 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.686650 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a709ba86-240f-4cdf-afc4-69a224583c98-config\") pod \"service-ca-operator-cf7c776b7-kkr78\" (UID: \"a709ba86-240f-4cdf-afc4-69a224583c98\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.686749 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.686675 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scpvl\" (UniqueName: \"kubernetes.io/projected/a709ba86-240f-4cdf-afc4-69a224583c98-kube-api-access-scpvl\") pod \"service-ca-operator-cf7c776b7-kkr78\" (UID: \"a709ba86-240f-4cdf-afc4-69a224583c98\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.772912 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.772889 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-cqvgw"] Sep 11 15:09:21.774983 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.774947 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.775923 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.775903 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm"] Sep 11 15:09:21.777380 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.777361 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x76rk\"" Sep 11 15:09:21.777596 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.777580 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:09:21.777674 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.777608 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 15:09:21.777674 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.777616 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 15:09:21.777674 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.777665 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 15:09:21.777974 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.777959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:21.780024 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.780006 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wn8jl\"" Sep 11 15:09:21.780310 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.780295 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 15:09:21.780383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.780311 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 15:09:21.780383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.780294 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 11 15:09:21.781794 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.781753 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 11 15:09:21.783178 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.783162 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 15:09:21.785141 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.785121 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-cqvgw"] Sep 11 15:09:21.787293 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.787271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a709ba86-240f-4cdf-afc4-69a224583c98-serving-cert\") pod \"service-ca-operator-cf7c776b7-kkr78\" (UID: \"a709ba86-240f-4cdf-afc4-69a224583c98\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.787391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.787304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a709ba86-240f-4cdf-afc4-69a224583c98-config\") pod \"service-ca-operator-cf7c776b7-kkr78\" (UID: \"a709ba86-240f-4cdf-afc4-69a224583c98\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.787391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.787330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpvl\" (UniqueName: \"kubernetes.io/projected/a709ba86-240f-4cdf-afc4-69a224583c98-kube-api-access-scpvl\") pod \"service-ca-operator-cf7c776b7-kkr78\" (UID: \"a709ba86-240f-4cdf-afc4-69a224583c98\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.788167 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.788077 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a709ba86-240f-4cdf-afc4-69a224583c98-config\") pod \"service-ca-operator-cf7c776b7-kkr78\" (UID: \"a709ba86-240f-4cdf-afc4-69a224583c98\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.789168 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.789141 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm"] Sep 11 15:09:21.791588 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.791568 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a709ba86-240f-4cdf-afc4-69a224583c98-serving-cert\") pod \"service-ca-operator-cf7c776b7-kkr78\" (UID: \"a709ba86-240f-4cdf-afc4-69a224583c98\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.799725 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.799701 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scpvl\" (UniqueName: \"kubernetes.io/projected/a709ba86-240f-4cdf-afc4-69a224583c98-kube-api-access-scpvl\") pod \"service-ca-operator-cf7c776b7-kkr78\" (UID: \"a709ba86-240f-4cdf-afc4-69a224583c98\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.887888 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.887837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43b10749-31c5-435a-bbe8-6245ed5bad1a-trusted-ca\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.887888 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.887865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrgdc\" (UniqueName: \"kubernetes.io/projected/43b10749-31c5-435a-bbe8-6245ed5bad1a-kube-api-access-nrgdc\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.888032 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.887886 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:21.888032 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.887906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43b10749-31c5-435a-bbe8-6245ed5bad1a-config\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.888032 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.887938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48b6\" (UniqueName: \"kubernetes.io/projected/466e17b5-124c-4a5d-8747-98696ecc7701-kube-api-access-g48b6\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:21.888032 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.887970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43b10749-31c5-435a-bbe8-6245ed5bad1a-serving-cert\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.888032 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.887987 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/466e17b5-124c-4a5d-8747-98696ecc7701-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:21.893871 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.893835 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" Sep 11 15:09:21.989340 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.989189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:21.989462 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:21.989345 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:21.989462 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.989348 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43b10749-31c5-435a-bbe8-6245ed5bad1a-config\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.989462 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:21.989421 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls podName:466e17b5-124c-4a5d-8747-98696ecc7701 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:22.489398885 +0000 UTC m=+44.534723852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-nbpfm" (UID: "466e17b5-124c-4a5d-8747-98696ecc7701") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:21.989462 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.989457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48b6\" (UniqueName: \"kubernetes.io/projected/466e17b5-124c-4a5d-8747-98696ecc7701-kube-api-access-g48b6\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:21.989681 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.989498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43b10749-31c5-435a-bbe8-6245ed5bad1a-serving-cert\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.989681 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.989524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/466e17b5-124c-4a5d-8747-98696ecc7701-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:21.989681 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.989650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43b10749-31c5-435a-bbe8-6245ed5bad1a-trusted-ca\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.989681 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.989676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrgdc\" (UniqueName: \"kubernetes.io/projected/43b10749-31c5-435a-bbe8-6245ed5bad1a-kube-api-access-nrgdc\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.990196 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.990175 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43b10749-31c5-435a-bbe8-6245ed5bad1a-config\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.990313 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.990298 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/466e17b5-124c-4a5d-8747-98696ecc7701-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:21.990735 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.990720 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43b10749-31c5-435a-bbe8-6245ed5bad1a-trusted-ca\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.992168 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.992151 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43b10749-31c5-435a-bbe8-6245ed5bad1a-serving-cert\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:21.997954 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.997928 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48b6\" (UniqueName: \"kubernetes.io/projected/466e17b5-124c-4a5d-8747-98696ecc7701-kube-api-access-g48b6\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:21.998416 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:21.998399 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrgdc\" (UniqueName: \"kubernetes.io/projected/43b10749-31c5-435a-bbe8-6245ed5bad1a-kube-api-access-nrgdc\") pod \"console-operator-696884fb9c-cqvgw\" (UID: \"43b10749-31c5-435a-bbe8-6245ed5bad1a\") " pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:22.009103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.009082 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c8f6b79cd-5c7gp"] Sep 11 15:09:22.012714 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.012701 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.015261 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.015229 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 15:09:22.015364 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.015262 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 15:09:22.015472 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.015457 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 15:09:22.015557 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.015542 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tz2qn\"" Sep 11 15:09:22.023630 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.023603 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 15:09:22.024570 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.024548 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c8f6b79cd-5c7gp"] Sep 11 15:09:22.085872 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.085848 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:22.090951 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.090935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-image-registry-private-configuration\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.091010 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.090959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.091010 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.090983 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-installation-pull-secrets\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.091089 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.091069 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-bound-sa-token\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.091124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.091103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-ca-trust-extracted\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.091163 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.091124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-certificates\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.091163 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.091141 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-trusted-ca\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.091163 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.091159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9vvl\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-kube-api-access-p9vvl\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.166580 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.166500 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78"] Sep 11 15:09:22.169711 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:22.169686 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda709ba86_240f_4cdf_afc4_69a224583c98.slice/crio-34f8612cac37e12e6a92826df63ee11a1369bdc7b0398495b7fc6070d2f822f4 WatchSource:0}: Error finding container 34f8612cac37e12e6a92826df63ee11a1369bdc7b0398495b7fc6070d2f822f4: Status 404 returned error can't find the container with id 34f8612cac37e12e6a92826df63ee11a1369bdc7b0398495b7fc6070d2f822f4 Sep 11 15:09:22.191860 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.191840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-ca-trust-extracted\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.191968 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.191874 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-certificates\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.191968 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.191898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-trusted-ca\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.191968 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.191927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9vvl\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-kube-api-access-p9vvl\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.192118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.192022 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-image-registry-private-configuration\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.192118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.192049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.192118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.192092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-installation-pull-secrets\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.192256 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.192151 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-bound-sa-token\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.192256 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:22.192220 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:22.192256 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:22.192242 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f6b79cd-5c7gp: secret "image-registry-tls" not found Sep 11 15:09:22.192391 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:22.192296 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls podName:8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:22.692276592 +0000 UTC m=+44.737601564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls") pod "image-registry-5c8f6b79cd-5c7gp" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c") : secret "image-registry-tls" not found Sep 11 15:09:22.192391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.192219 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-ca-trust-extracted\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.192554 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.192506 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-certificates\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.193190 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.193130 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-trusted-ca\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.195096 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.195073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-installation-pull-secrets\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.195167 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.195073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-image-registry-private-configuration\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.201246 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.201223 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-bound-sa-token\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.201605 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.201588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9vvl\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-kube-api-access-p9vvl\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.351851 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.351823 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-cqvgw"] Sep 11 15:09:22.354941 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:22.354917 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b10749_31c5_435a_bbe8_6245ed5bad1a.slice/crio-04dffdc60d10f1be0687b60d38d7d63bb7a4b00bee267dab4aae493cbf6fd685 WatchSource:0}: Error finding container 04dffdc60d10f1be0687b60d38d7d63bb7a4b00bee267dab4aae493cbf6fd685: Status 404 returned error can't find the container with id 04dffdc60d10f1be0687b60d38d7d63bb7a4b00bee267dab4aae493cbf6fd685 Sep 11 15:09:22.493958 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.493903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:22.494049 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:22.493994 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:22.494049 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:22.494041 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls podName:466e17b5-124c-4a5d-8747-98696ecc7701 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:23.494027875 +0000 UTC m=+45.539352833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-nbpfm" (UID: "466e17b5-124c-4a5d-8747-98696ecc7701") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:22.695952 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.695927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:22.696274 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:22.696051 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:22.696274 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:22.696063 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f6b79cd-5c7gp: secret "image-registry-tls" not found Sep 11 15:09:22.696274 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:22.696126 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls podName:8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:23.696111957 +0000 UTC m=+45.741436916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls") pod "image-registry-5c8f6b79cd-5c7gp" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c") : secret "image-registry-tls" not found Sep 11 15:09:22.760431 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.760364 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" event={"ID":"43b10749-31c5-435a-bbe8-6245ed5bad1a","Type":"ContainerStarted","Data":"04dffdc60d10f1be0687b60d38d7d63bb7a4b00bee267dab4aae493cbf6fd685"} Sep 11 15:09:22.761512 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:22.761492 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" event={"ID":"a709ba86-240f-4cdf-afc4-69a224583c98","Type":"ContainerStarted","Data":"34f8612cac37e12e6a92826df63ee11a1369bdc7b0398495b7fc6070d2f822f4"} Sep 11 15:09:23.099739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:23.099705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:23.099924 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:23.099875 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:23.099991 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:23.099951 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls podName:07bd3b2e-2168-40ec-8fcf-6a55bfa7468c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:31.099930126 +0000 UTC m=+53.145255085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls") pod "dns-default-p94fh" (UID: "07bd3b2e-2168-40ec-8fcf-6a55bfa7468c") : secret "dns-default-metrics-tls" not found Sep 11 15:09:23.502419 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:23.502338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:23.502581 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:23.502493 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:23.502581 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:23.502561 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls podName:466e17b5-124c-4a5d-8747-98696ecc7701 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:25.502545233 +0000 UTC m=+47.547870196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-nbpfm" (UID: "466e17b5-124c-4a5d-8747-98696ecc7701") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:23.704201 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:23.704165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:23.704644 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:23.704341 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:23.704644 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:23.704360 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f6b79cd-5c7gp: secret "image-registry-tls" not found Sep 11 15:09:23.704644 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:23.704425 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls podName:8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:25.704405793 +0000 UTC m=+47.749730750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls") pod "image-registry-5c8f6b79cd-5c7gp" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c") : secret "image-registry-tls" not found Sep 11 15:09:24.769194 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:24.769162 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/0.log" Sep 11 15:09:24.769629 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:24.769205 2573 generic.go:356] "Generic (PLEG): container finished" podID="43b10749-31c5-435a-bbe8-6245ed5bad1a" containerID="1012d20c98091ff9521e3b50f2d16760e0e63623a4ecd13b55c7acca6cd795ec" exitCode=255 Sep 11 15:09:24.769629 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:24.769238 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" event={"ID":"43b10749-31c5-435a-bbe8-6245ed5bad1a","Type":"ContainerDied","Data":"1012d20c98091ff9521e3b50f2d16760e0e63623a4ecd13b55c7acca6cd795ec"} Sep 11 15:09:24.769629 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:24.769547 2573 scope.go:117] "RemoveContainer" containerID="1012d20c98091ff9521e3b50f2d16760e0e63623a4ecd13b55c7acca6cd795ec" Sep 11 15:09:25.515525 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:25.515447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:25.515682 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:25.515593 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:25.515682 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:25.515655 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls podName:466e17b5-124c-4a5d-8747-98696ecc7701 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:29.51563997 +0000 UTC m=+51.560964928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-nbpfm" (UID: "466e17b5-124c-4a5d-8747-98696ecc7701") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:25.716898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:25.716870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:25.717024 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:25.717009 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:25.717060 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:25.717027 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f6b79cd-5c7gp: secret "image-registry-tls" not found Sep 11 15:09:25.717102 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:25.717078 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls podName:8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:29.717063637 +0000 UTC m=+51.762388598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls") pod "image-registry-5c8f6b79cd-5c7gp" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c") : secret "image-registry-tls" not found Sep 11 15:09:25.773726 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:25.773677 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:09:25.774171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:25.774154 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/0.log" Sep 11 15:09:25.774252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:25.774195 2573 generic.go:356] "Generic (PLEG): container finished" podID="43b10749-31c5-435a-bbe8-6245ed5bad1a" containerID="c3a9005f6fe2f6ebaeb76b39809219ccda808dd1cb35908f55f545e0908edf89" exitCode=255 Sep 11 15:09:25.774306 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:25.774282 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" event={"ID":"43b10749-31c5-435a-bbe8-6245ed5bad1a","Type":"ContainerDied","Data":"c3a9005f6fe2f6ebaeb76b39809219ccda808dd1cb35908f55f545e0908edf89"} Sep 11 15:09:25.774402 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:25.774385 2573 scope.go:117] "RemoveContainer" containerID="1012d20c98091ff9521e3b50f2d16760e0e63623a4ecd13b55c7acca6cd795ec" Sep 11 15:09:25.774614 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:25.774600 2573 scope.go:117] "RemoveContainer" containerID="c3a9005f6fe2f6ebaeb76b39809219ccda808dd1cb35908f55f545e0908edf89" Sep 11 15:09:25.774796 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:25.774758 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-cqvgw_openshift-console-operator(43b10749-31c5-435a-bbe8-6245ed5bad1a)\"" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" podUID="43b10749-31c5-435a-bbe8-6245ed5bad1a" Sep 11 15:09:25.775965 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:25.775902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" event={"ID":"a709ba86-240f-4cdf-afc4-69a224583c98","Type":"ContainerStarted","Data":"52af77789376eb83d85254195d5bfecd4f4f19b94506a5cebd717606bc02b623"} Sep 11 15:09:25.805448 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:25.805407 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" podStartSLOduration=1.831873724 podStartE2EDuration="4.805395099s" totalImagesPullingTime="2.973521375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:25.805138126 +0000 UTC m=+47.850463105" watchObservedRunningTime="2026-09-11 15:09:25.805395099 +0000 UTC m=+47.850720080" Sep 11 15:09:26.539326 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.539299 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xdncs"] Sep 11 15:09:26.543571 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.543556 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.546482 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.546459 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 15:09:26.547004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.546985 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cr68x\"" Sep 11 15:09:26.547462 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.547444 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 15:09:26.547524 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.547488 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 15:09:26.550104 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.549792 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 15:09:26.566249 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.566228 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xdncs"] Sep 11 15:09:26.623030 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.623010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.623131 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.623038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs96n\" (UniqueName: \"kubernetes.io/projected/fd2de11f-c409-4a82-af57-9883c2e14767-kube-api-access-fs96n\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.623131 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.623087 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.623131 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.623117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd2de11f-c409-4a82-af57-9883c2e14767-crio-socket\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.623229 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.623145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd2de11f-c409-4a82-af57-9883c2e14767-data-volume\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.724296 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.724274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs96n\" (UniqueName: \"kubernetes.io/projected/fd2de11f-c409-4a82-af57-9883c2e14767-kube-api-access-fs96n\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.724448 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.724317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.724448 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.724349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd2de11f-c409-4a82-af57-9883c2e14767-crio-socket\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.724448 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.724376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd2de11f-c409-4a82-af57-9883c2e14767-data-volume\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.724448 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:26.724405 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:09:26.724593 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.724436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.724593 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:26.724475 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm podName:fd2de11f-c409-4a82-af57-9883c2e14767 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:27.224454877 +0000 UTC m=+49.269779859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xdncs" (UID: "fd2de11f-c409-4a82-af57-9883c2e14767") : configmap "kube-rbac-proxy" not found Sep 11 15:09:26.724593 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:26.724557 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:26.724691 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:26.724603 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls podName:fd2de11f-c409-4a82-af57-9883c2e14767 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:27.224591675 +0000 UTC m=+49.269916634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xdncs" (UID: "fd2de11f-c409-4a82-af57-9883c2e14767") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:26.724691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.724597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd2de11f-c409-4a82-af57-9883c2e14767-crio-socket\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.724757 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.724715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd2de11f-c409-4a82-af57-9883c2e14767-data-volume\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.733291 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.733264 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs96n\" (UniqueName: \"kubernetes.io/projected/fd2de11f-c409-4a82-af57-9883c2e14767-kube-api-access-fs96n\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:26.779867 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.779851 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:09:26.780159 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:26.780117 2573 scope.go:117] "RemoveContainer" containerID="c3a9005f6fe2f6ebaeb76b39809219ccda808dd1cb35908f55f545e0908edf89" Sep 11 15:09:26.780250 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:26.780231 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-cqvgw_openshift-console-operator(43b10749-31c5-435a-bbe8-6245ed5bad1a)\"" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" podUID="43b10749-31c5-435a-bbe8-6245ed5bad1a" Sep 11 15:09:27.229165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:27.229141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:27.229350 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:27.229187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:27.229350 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:27.229290 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:09:27.229350 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:27.229296 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:27.229350 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:27.229344 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm podName:fd2de11f-c409-4a82-af57-9883c2e14767 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:28.229330098 +0000 UTC m=+50.274655056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xdncs" (UID: "fd2de11f-c409-4a82-af57-9883c2e14767") : configmap "kube-rbac-proxy" not found Sep 11 15:09:27.229503 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:27.229358 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls podName:fd2de11f-c409-4a82-af57-9883c2e14767 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:28.229351397 +0000 UTC m=+50.274676355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xdncs" (UID: "fd2de11f-c409-4a82-af57-9883c2e14767") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:28.235259 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:28.235222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:28.235653 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:28.235285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:28.235653 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:28.235349 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:09:28.235653 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:28.235371 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:28.235653 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:28.235420 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm podName:fd2de11f-c409-4a82-af57-9883c2e14767 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:30.235404646 +0000 UTC m=+52.280729603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xdncs" (UID: "fd2de11f-c409-4a82-af57-9883c2e14767") : configmap "kube-rbac-proxy" not found Sep 11 15:09:28.235653 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:28.235437 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls podName:fd2de11f-c409-4a82-af57-9883c2e14767 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:30.235429991 +0000 UTC m=+52.280754948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xdncs" (UID: "fd2de11f-c409-4a82-af57-9883c2e14767") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:29.492533 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:29.492505 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tjhlf_1813d8d4-743c-4b24-b861-aa702a8f5e5b/dns-node-resolver/0.log" Sep 11 15:09:29.543963 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:29.543923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:29.544148 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:29.544101 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:29.544214 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:29.544184 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls podName:466e17b5-124c-4a5d-8747-98696ecc7701 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:37.544162686 +0000 UTC m=+59.589487675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-nbpfm" (UID: "466e17b5-124c-4a5d-8747-98696ecc7701") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:29.745474 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:29.745406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:29.745624 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:29.745546 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:29.745624 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:29.745563 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8f6b79cd-5c7gp: secret "image-registry-tls" not found Sep 11 15:09:29.745624 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:29.745620 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls podName:8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:37.745604645 +0000 UTC m=+59.790929608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls") pod "image-registry-5c8f6b79cd-5c7gp" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c") : secret "image-registry-tls" not found Sep 11 15:09:30.250607 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:30.250571 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:30.250803 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:30.250696 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:09:30.250803 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:30.250704 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:30.250803 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:30.250759 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm podName:fd2de11f-c409-4a82-af57-9883c2e14767 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:34.250742148 +0000 UTC m=+56.296067106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xdncs" (UID: "fd2de11f-c409-4a82-af57-9883c2e14767") : configmap "kube-rbac-proxy" not found Sep 11 15:09:30.250997 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:30.250871 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:30.250997 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:30.250951 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls podName:fd2de11f-c409-4a82-af57-9883c2e14767 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:34.250932629 +0000 UTC m=+56.296257590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xdncs" (UID: "fd2de11f-c409-4a82-af57-9883c2e14767") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:30.490005 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:30.489974 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r4zv8_3bcac49b-9179-411c-9dd7-1de6585a7a0f/node-ca/0.log" Sep 11 15:09:31.156690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:31.156655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:31.157175 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:31.156846 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:31.157175 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:31.156927 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls podName:07bd3b2e-2168-40ec-8fcf-6a55bfa7468c nodeName:}" failed. No retries permitted until 2026-09-11 15:09:47.156906848 +0000 UTC m=+69.202231812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls") pod "dns-default-p94fh" (UID: "07bd3b2e-2168-40ec-8fcf-6a55bfa7468c") : secret "dns-default-metrics-tls" not found Sep 11 15:09:31.289907 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:31.289875 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-smjz7_522b2527-f838-4d7d-95cf-6f2ce8ff4bd4/serve-healthcheck-canary/0.log" Sep 11 15:09:32.086664 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:32.086632 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:32.087034 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:32.086687 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:32.087100 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:32.087078 2573 scope.go:117] "RemoveContainer" containerID="c3a9005f6fe2f6ebaeb76b39809219ccda808dd1cb35908f55f545e0908edf89" Sep 11 15:09:32.087289 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:32.087266 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-cqvgw_openshift-console-operator(43b10749-31c5-435a-bbe8-6245ed5bad1a)\"" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" podUID="43b10749-31c5-435a-bbe8-6245ed5bad1a" Sep 11 15:09:34.280452 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:34.280415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:34.280924 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:34.280502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:34.280924 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:34.280537 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:09:34.280924 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:34.280602 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm podName:fd2de11f-c409-4a82-af57-9883c2e14767 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:42.280587266 +0000 UTC m=+64.325912229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xdncs" (UID: "fd2de11f-c409-4a82-af57-9883c2e14767") : configmap "kube-rbac-proxy" not found Sep 11 15:09:34.280924 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:34.280606 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:34.280924 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:34.280646 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls podName:fd2de11f-c409-4a82-af57-9883c2e14767 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:42.280633915 +0000 UTC m=+64.325958874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xdncs" (UID: "fd2de11f-c409-4a82-af57-9883c2e14767") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:37.604710 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:37.604674 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:37.605174 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:37.604857 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:37.605174 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:09:37.604945 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls podName:466e17b5-124c-4a5d-8747-98696ecc7701 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:53.604925017 +0000 UTC m=+75.650249989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-nbpfm" (UID: "466e17b5-124c-4a5d-8747-98696ecc7701") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:09:37.806150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:37.806124 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:37.808599 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:37.808569 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls\") pod \"image-registry-5c8f6b79cd-5c7gp\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:37.924991 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:37.924931 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:38.199511 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:38.199438 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c8f6b79cd-5c7gp"] Sep 11 15:09:38.202456 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:38.202425 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f88c4ef_6848_4c12_8f42_c4bdb2d9a46c.slice/crio-2b6a2a8d680248aff97c331038b404c9b23b1a8eed298c634d89b0a2dfca8ed5 WatchSource:0}: Error finding container 2b6a2a8d680248aff97c331038b404c9b23b1a8eed298c634d89b0a2dfca8ed5: Status 404 returned error can't find the container with id 2b6a2a8d680248aff97c331038b404c9b23b1a8eed298c634d89b0a2dfca8ed5 Sep 11 15:09:38.809975 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:38.809939 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" event={"ID":"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c","Type":"ContainerStarted","Data":"fd234a17c8e5bbf34acc3707ab44789cb2559056791cd5841cc229550049151f"} Sep 11 15:09:38.810334 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:38.809987 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" event={"ID":"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c","Type":"ContainerStarted","Data":"2b6a2a8d680248aff97c331038b404c9b23b1a8eed298c634d89b0a2dfca8ed5"} Sep 11 15:09:38.810334 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:38.810083 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:09:38.828970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:38.828372 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" podStartSLOduration=17.828340781 podStartE2EDuration="17.828340781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:38.827975643 +0000 UTC m=+60.873300622" watchObservedRunningTime="2026-09-11 15:09:38.828340781 +0000 UTC m=+60.873665761" Sep 11 15:09:41.739465 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:41.739440 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlrgc" Sep 11 15:09:42.339093 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:42.339060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:42.339273 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:42.339120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:42.339668 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:42.339643 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd2de11f-c409-4a82-af57-9883c2e14767-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:42.341545 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:42.341526 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd2de11f-c409-4a82-af57-9883c2e14767-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdncs\" (UID: \"fd2de11f-c409-4a82-af57-9883c2e14767\") " pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:42.455372 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:42.455347 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cr68x\"" Sep 11 15:09:42.463489 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:42.463473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xdncs" Sep 11 15:09:42.743233 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:42.743210 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xdncs"] Sep 11 15:09:42.746042 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:42.746017 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2de11f_c409_4a82_af57_9883c2e14767.slice/crio-f9ca54250962a8f7b52175f4bb189857541958318edddaa5a6c617d45492a5b8 WatchSource:0}: Error finding container f9ca54250962a8f7b52175f4bb189857541958318edddaa5a6c617d45492a5b8: Status 404 returned error can't find the container with id f9ca54250962a8f7b52175f4bb189857541958318edddaa5a6c617d45492a5b8 Sep 11 15:09:42.821640 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:42.821613 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xdncs" event={"ID":"fd2de11f-c409-4a82-af57-9883c2e14767","Type":"ContainerStarted","Data":"b00de924776b816cec7fb9f33f2c85c6ba960228cc8d894702276d22434aa0e4"} Sep 11 15:09:42.821764 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:42.821666 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xdncs" event={"ID":"fd2de11f-c409-4a82-af57-9883c2e14767","Type":"ContainerStarted","Data":"f9ca54250962a8f7b52175f4bb189857541958318edddaa5a6c617d45492a5b8"} Sep 11 15:09:43.245295 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.245270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:43.245466 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.245304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:43.247887 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.247866 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 15:09:43.247963 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.247872 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 15:09:43.258460 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.258441 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96e306-761e-4985-b93d-2d8fab445fe1-metrics-certs\") pod \"network-metrics-daemon-m7drl\" (UID: \"9d96e306-761e-4985-b93d-2d8fab445fe1\") " pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:43.258532 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.258476 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f504ee5-37b1-4ba9-8331-92ce973dc316-original-pull-secret\") pod \"global-pull-secret-syncer-c6p9x\" (UID: \"2f504ee5-37b1-4ba9-8331-92ce973dc316\") " pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:43.346422 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.346393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kd67\" (UniqueName: \"kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67\") pod \"network-check-target-7z2zb\" (UID: \"a1646ede-0a03-43a1-8f28-873a9ac43905\") " pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:43.349486 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.349468 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 15:09:43.359764 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.359750 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 15:09:43.370470 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.370447 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kd67\" (UniqueName: \"kubernetes.io/projected/a1646ede-0a03-43a1-8f28-873a9ac43905-kube-api-access-6kd67\") pod \"network-check-target-7z2zb\" (UID: \"a1646ede-0a03-43a1-8f28-873a9ac43905\") " pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:43.397869 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.397845 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4h8rg\"" Sep 11 15:09:43.401813 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.401798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6p9x" Sep 11 15:09:43.406053 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.406030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m7drl" Sep 11 15:09:43.409281 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.408144 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2f6h7\"" Sep 11 15:09:43.416965 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.416947 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:43.803330 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.803146 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m7drl"] Sep 11 15:09:43.806542 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.806517 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c6p9x"] Sep 11 15:09:43.807435 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:43.807416 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7z2zb"] Sep 11 15:09:43.873867 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:43.873838 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d96e306_761e_4985_b93d_2d8fab445fe1.slice/crio-697a4e9edbd9f3480570c95dc845cf6fc97caa65ab1e1feaa498cd9697944e08 WatchSource:0}: Error finding container 697a4e9edbd9f3480570c95dc845cf6fc97caa65ab1e1feaa498cd9697944e08: Status 404 returned error can't find the container with id 697a4e9edbd9f3480570c95dc845cf6fc97caa65ab1e1feaa498cd9697944e08 Sep 11 15:09:43.874550 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:43.874518 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f504ee5_37b1_4ba9_8331_92ce973dc316.slice/crio-45b2843aa99a835f3c88426aace35d19e87b8e2462dfe08d8876f88711158368 WatchSource:0}: Error finding container 45b2843aa99a835f3c88426aace35d19e87b8e2462dfe08d8876f88711158368: Status 404 returned error can't find the container with id 45b2843aa99a835f3c88426aace35d19e87b8e2462dfe08d8876f88711158368 Sep 11 15:09:43.875223 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:43.875173 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1646ede_0a03_43a1_8f28_873a9ac43905.slice/crio-81fc011e937e812b90b4475a37e67c7dafb57481ae6920e71f42d7426b44a6e6 WatchSource:0}: Error finding container 81fc011e937e812b90b4475a37e67c7dafb57481ae6920e71f42d7426b44a6e6: Status 404 returned error can't find the container with id 81fc011e937e812b90b4475a37e67c7dafb57481ae6920e71f42d7426b44a6e6 Sep 11 15:09:44.832253 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:44.832176 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xdncs" event={"ID":"fd2de11f-c409-4a82-af57-9883c2e14767","Type":"ContainerStarted","Data":"172b6ec160b20d7579ec82b7329104e55bddbbe699719d3d21f5f12cba0f73b2"} Sep 11 15:09:44.833511 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:44.833475 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c6p9x" event={"ID":"2f504ee5-37b1-4ba9-8331-92ce973dc316","Type":"ContainerStarted","Data":"45b2843aa99a835f3c88426aace35d19e87b8e2462dfe08d8876f88711158368"} Sep 11 15:09:44.835349 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:44.835317 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m7drl" event={"ID":"9d96e306-761e-4985-b93d-2d8fab445fe1","Type":"ContainerStarted","Data":"697a4e9edbd9f3480570c95dc845cf6fc97caa65ab1e1feaa498cd9697944e08"} Sep 11 15:09:44.837421 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:44.837368 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7z2zb" event={"ID":"a1646ede-0a03-43a1-8f28-873a9ac43905","Type":"ContainerStarted","Data":"81fc011e937e812b90b4475a37e67c7dafb57481ae6920e71f42d7426b44a6e6"} Sep 11 15:09:45.844939 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:45.844901 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m7drl" event={"ID":"9d96e306-761e-4985-b93d-2d8fab445fe1","Type":"ContainerStarted","Data":"f78620faadf82426f12ddd55cb4a18797b3e70cd6c312d8f29298bbe82f2227a"} Sep 11 15:09:45.845354 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:45.844969 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m7drl" event={"ID":"9d96e306-761e-4985-b93d-2d8fab445fe1","Type":"ContainerStarted","Data":"99870f2b79791a546e67aa51807d6844f556a9afbc9fadd7fefa6da547b34e8d"} Sep 11 15:09:45.861292 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:45.861237 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m7drl" podStartSLOduration=66.526327346 podStartE2EDuration="1m7.861216857s" totalImagesPullingTime="1.334889511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:45.860705918 +0000 UTC m=+67.906030898" watchObservedRunningTime="2026-09-11 15:09:45.861216857 +0000 UTC m=+67.906541838" Sep 11 15:09:46.487430 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:46.487394 2573 scope.go:117] "RemoveContainer" containerID="c3a9005f6fe2f6ebaeb76b39809219ccda808dd1cb35908f55f545e0908edf89" Sep 11 15:09:46.851002 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:46.850960 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xdncs" event={"ID":"fd2de11f-c409-4a82-af57-9883c2e14767","Type":"ContainerStarted","Data":"03494d0166ca03289bedcbe4abf96de8bb71d58c13dcb5ad5a63b3c288023b16"} Sep 11 15:09:46.869109 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:46.869056 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xdncs" podStartSLOduration=17.813057663 podStartE2EDuration="20.869035752s" totalImagesPullingTime="3.055978089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:09:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:46.868531339 +0000 UTC m=+68.913856319" watchObservedRunningTime="2026-09-11 15:09:46.869035752 +0000 UTC m=+68.914360733" Sep 11 15:09:47.179498 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:47.179404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:47.182118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:47.182088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07bd3b2e-2168-40ec-8fcf-6a55bfa7468c-metrics-tls\") pod \"dns-default-p94fh\" (UID: \"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c\") " pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:47.435612 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:47.435534 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zldr7\"" Sep 11 15:09:47.443800 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:47.443759 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:48.765068 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.765041 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4"] Sep 11 15:09:48.767891 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.767873 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:48.770321 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.770301 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 15:09:48.771023 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.771005 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 15:09:48.771108 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.771007 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 11 15:09:48.771108 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.771047 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 15:09:48.782572 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.782551 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4"] Sep 11 15:09:48.787695 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.787669 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/811f5e71-6594-476d-b384-d13235158103-tmp\") pod \"klusterlet-addon-workmgr-56f749c75b-dscc4\" (UID: \"811f5e71-6594-476d-b384-d13235158103\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:48.787823 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.787723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfmqp\" (UniqueName: \"kubernetes.io/projected/811f5e71-6594-476d-b384-d13235158103-kube-api-access-dfmqp\") pod \"klusterlet-addon-workmgr-56f749c75b-dscc4\" (UID: \"811f5e71-6594-476d-b384-d13235158103\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:48.787823 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.787812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/811f5e71-6594-476d-b384-d13235158103-klusterlet-config\") pod \"klusterlet-addon-workmgr-56f749c75b-dscc4\" (UID: \"811f5e71-6594-476d-b384-d13235158103\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:48.811673 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.809341 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c8f6b79cd-5c7gp"] Sep 11 15:09:48.843938 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.843914 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78d65bbf48-lstr8"] Sep 11 15:09:48.852884 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.852863 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.867137 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.866733 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78d65bbf48-lstr8"] Sep 11 15:09:48.889852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67eef969-04fa-4cf5-a5e6-22b3cb066888-bound-sa-token\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.889852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/811f5e71-6594-476d-b384-d13235158103-tmp\") pod \"klusterlet-addon-workmgr-56f749c75b-dscc4\" (UID: \"811f5e71-6594-476d-b384-d13235158103\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:48.889852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67eef969-04fa-4cf5-a5e6-22b3cb066888-image-registry-private-configuration\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.889852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889560 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67eef969-04fa-4cf5-a5e6-22b3cb066888-installation-pull-secrets\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.889852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889593 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67eef969-04fa-4cf5-a5e6-22b3cb066888-registry-certificates\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.889852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889616 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfmqp\" (UniqueName: \"kubernetes.io/projected/811f5e71-6594-476d-b384-d13235158103-kube-api-access-dfmqp\") pod \"klusterlet-addon-workmgr-56f749c75b-dscc4\" (UID: \"811f5e71-6594-476d-b384-d13235158103\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:48.889852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889645 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/811f5e71-6594-476d-b384-d13235158103-klusterlet-config\") pod \"klusterlet-addon-workmgr-56f749c75b-dscc4\" (UID: \"811f5e71-6594-476d-b384-d13235158103\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:48.889852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889675 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67eef969-04fa-4cf5-a5e6-22b3cb066888-trusted-ca\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.889852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889727 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67eef969-04fa-4cf5-a5e6-22b3cb066888-registry-tls\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.890330 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889868 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/811f5e71-6594-476d-b384-d13235158103-tmp\") pod \"klusterlet-addon-workmgr-56f749c75b-dscc4\" (UID: \"811f5e71-6594-476d-b384-d13235158103\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:48.890532 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.889765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67eef969-04fa-4cf5-a5e6-22b3cb066888-ca-trust-extracted\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.890532 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.890444 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5fq\" (UniqueName: \"kubernetes.io/projected/67eef969-04fa-4cf5-a5e6-22b3cb066888-kube-api-access-kd5fq\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.892540 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.892519 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/811f5e71-6594-476d-b384-d13235158103-klusterlet-config\") pod \"klusterlet-addon-workmgr-56f749c75b-dscc4\" (UID: \"811f5e71-6594-476d-b384-d13235158103\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:48.902685 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.902660 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfmqp\" (UniqueName: \"kubernetes.io/projected/811f5e71-6594-476d-b384-d13235158103-kube-api-access-dfmqp\") pod \"klusterlet-addon-workmgr-56f749c75b-dscc4\" (UID: \"811f5e71-6594-476d-b384-d13235158103\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:48.991526 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.991461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67eef969-04fa-4cf5-a5e6-22b3cb066888-registry-certificates\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.991526 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.991505 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67eef969-04fa-4cf5-a5e6-22b3cb066888-trusted-ca\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.991719 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.991552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67eef969-04fa-4cf5-a5e6-22b3cb066888-registry-tls\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.991719 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.991683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67eef969-04fa-4cf5-a5e6-22b3cb066888-ca-trust-extracted\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.991852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.991725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5fq\" (UniqueName: \"kubernetes.io/projected/67eef969-04fa-4cf5-a5e6-22b3cb066888-kube-api-access-kd5fq\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.991852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.991798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67eef969-04fa-4cf5-a5e6-22b3cb066888-bound-sa-token\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.991852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.991840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67eef969-04fa-4cf5-a5e6-22b3cb066888-image-registry-private-configuration\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.991990 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.991867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67eef969-04fa-4cf5-a5e6-22b3cb066888-installation-pull-secrets\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.992405 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.992349 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67eef969-04fa-4cf5-a5e6-22b3cb066888-ca-trust-extracted\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.992938 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.992643 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67eef969-04fa-4cf5-a5e6-22b3cb066888-registry-certificates\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.992938 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.992745 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67eef969-04fa-4cf5-a5e6-22b3cb066888-trusted-ca\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.994121 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.994100 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67eef969-04fa-4cf5-a5e6-22b3cb066888-registry-tls\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.994386 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.994367 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67eef969-04fa-4cf5-a5e6-22b3cb066888-image-registry-private-configuration\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:48.995184 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:48.995159 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67eef969-04fa-4cf5-a5e6-22b3cb066888-installation-pull-secrets\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:49.002319 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.002295 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5fq\" (UniqueName: \"kubernetes.io/projected/67eef969-04fa-4cf5-a5e6-22b3cb066888-kube-api-access-kd5fq\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:49.002400 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.002360 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67eef969-04fa-4cf5-a5e6-22b3cb066888-bound-sa-token\") pod \"image-registry-78d65bbf48-lstr8\" (UID: \"67eef969-04fa-4cf5-a5e6-22b3cb066888\") " pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:49.073235 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.073124 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p94fh"] Sep 11 15:09:49.076131 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:49.076103 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bd3b2e_2168_40ec_8fcf_6a55bfa7468c.slice/crio-79edc79473e269cfc126a7663541f35c4b137b42c9633d8ed51018c32ada752a WatchSource:0}: Error finding container 79edc79473e269cfc126a7663541f35c4b137b42c9633d8ed51018c32ada752a: Status 404 returned error can't find the container with id 79edc79473e269cfc126a7663541f35c4b137b42c9633d8ed51018c32ada752a Sep 11 15:09:49.077181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.077161 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:49.169939 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.169907 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:49.349467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.349439 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4"] Sep 11 15:09:49.352202 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:49.352180 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811f5e71_6594_476d_b384_d13235158103.slice/crio-b66190b0e764eeb1e71aafd5ba418992ed8540dee0689121f131ba219bd35e83 WatchSource:0}: Error finding container b66190b0e764eeb1e71aafd5ba418992ed8540dee0689121f131ba219bd35e83: Status 404 returned error can't find the container with id b66190b0e764eeb1e71aafd5ba418992ed8540dee0689121f131ba219bd35e83 Sep 11 15:09:49.451950 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.451926 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78d65bbf48-lstr8"] Sep 11 15:09:49.455120 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:49.455098 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67eef969_04fa_4cf5_a5e6_22b3cb066888.slice/crio-5e8562b035da528d2699cbbb3572770bbf44197cece91c1ba5ffb34c5bbcc1ee WatchSource:0}: Error finding container 5e8562b035da528d2699cbbb3572770bbf44197cece91c1ba5ffb34c5bbcc1ee: Status 404 returned error can't find the container with id 5e8562b035da528d2699cbbb3572770bbf44197cece91c1ba5ffb34c5bbcc1ee Sep 11 15:09:49.865337 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.865292 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" event={"ID":"67eef969-04fa-4cf5-a5e6-22b3cb066888","Type":"ContainerStarted","Data":"8f9243187ba3e84ade5b0bb526e1f43e78a367e3d073c7feea2a743c556ea127"} Sep 11 15:09:49.865809 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.865348 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" event={"ID":"67eef969-04fa-4cf5-a5e6-22b3cb066888","Type":"ContainerStarted","Data":"5e8562b035da528d2699cbbb3572770bbf44197cece91c1ba5ffb34c5bbcc1ee"} Sep 11 15:09:49.865809 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.865406 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:09:49.866817 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.866793 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p94fh" event={"ID":"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c","Type":"ContainerStarted","Data":"79edc79473e269cfc126a7663541f35c4b137b42c9633d8ed51018c32ada752a"} Sep 11 15:09:49.869050 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.869028 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c6p9x" event={"ID":"2f504ee5-37b1-4ba9-8331-92ce973dc316","Type":"ContainerStarted","Data":"8163a4aa536f504d155ce7e36dca4fcaa2aaac6a3e8a6dec7667e77372cbac22"} Sep 11 15:09:49.871241 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.871215 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7z2zb" event={"ID":"a1646ede-0a03-43a1-8f28-873a9ac43905","Type":"ContainerStarted","Data":"f5e93c92651e6f8c7030ea4972f05c56bc80ccd15bcc7a151949bc2ee4b23de0"} Sep 11 15:09:49.871362 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.871337 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:09:49.872761 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.872738 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" event={"ID":"811f5e71-6594-476d-b384-d13235158103","Type":"ContainerStarted","Data":"b66190b0e764eeb1e71aafd5ba418992ed8540dee0689121f131ba219bd35e83"} Sep 11 15:09:49.875003 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.874980 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:09:49.875118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.875023 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" event={"ID":"43b10749-31c5-435a-bbe8-6245ed5bad1a","Type":"ContainerStarted","Data":"065e1cc5543a13e446c21ffdbc156993f52a25dbf2e614e1c65538460fab428e"} Sep 11 15:09:49.875429 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.875411 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:49.918038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.917973 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" podStartSLOduration=1.917952817 podStartE2EDuration="1.917952817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:49.916611393 +0000 UTC m=+71.961936375" watchObservedRunningTime="2026-09-11 15:09:49.917952817 +0000 UTC m=+71.963277798" Sep 11 15:09:49.946375 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.946342 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" Sep 11 15:09:49.990398 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.990344 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7z2zb" podStartSLOduration=67.09402012 podStartE2EDuration="1m11.990329181s" totalImagesPullingTime="4.896309061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:49.963964838 +0000 UTC m=+72.009289819" watchObservedRunningTime="2026-09-11 15:09:49.990329181 +0000 UTC m=+72.035654188" Sep 11 15:09:49.990607 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:49.990574 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c6p9x" podStartSLOduration=67.087732174 podStartE2EDuration="1m11.99056487s" totalImagesPullingTime="4.902832696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:49.989714798 +0000 UTC m=+72.035039781" watchObservedRunningTime="2026-09-11 15:09:49.99056487 +0000 UTC m=+72.035889851" Sep 11 15:09:50.021539 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:50.021484 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-cqvgw" podStartSLOduration=27.111172724 podStartE2EDuration="29.021469131s" totalImagesPullingTime="1.910296407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:50.021068544 +0000 UTC m=+72.066393524" watchObservedRunningTime="2026-09-11 15:09:50.021469131 +0000 UTC m=+72.066794111" Sep 11 15:09:53.628221 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.628167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:53.630987 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.630963 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/466e17b5-124c-4a5d-8747-98696ecc7701-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-nbpfm\" (UID: \"466e17b5-124c-4a5d-8747-98696ecc7701\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:53.892838 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.892746 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p94fh" event={"ID":"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c","Type":"ContainerStarted","Data":"5658e3c792b3ccbfcc17afbcdc0e5464fec30a1435bb7b1db4eeade2fb89af5e"} Sep 11 15:09:53.892838 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.892819 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p94fh" event={"ID":"07bd3b2e-2168-40ec-8fcf-6a55bfa7468c","Type":"ContainerStarted","Data":"3c442087ad7011f053383cc183d9d638722d157717d39f0926b6c9d097a47fe5"} Sep 11 15:09:53.893061 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.893039 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p94fh" Sep 11 15:09:53.893897 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.893879 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wn8jl\"" Sep 11 15:09:53.894383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.894355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" event={"ID":"811f5e71-6594-476d-b384-d13235158103","Type":"ContainerStarted","Data":"f896c00cf6c3eadfea0bc82b6d4d97b3ba0b68aa1105d99e6f9504e6d46fb392"} Sep 11 15:09:53.894475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.894462 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:53.896142 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.896125 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" Sep 11 15:09:53.901834 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.901817 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" Sep 11 15:09:53.911619 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.911574 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p94fh" podStartSLOduration=34.688413292 podStartE2EDuration="38.911558241s" totalImagesPullingTime="4.223144949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:53.91052545 +0000 UTC m=+75.955850431" watchObservedRunningTime="2026-09-11 15:09:53.911558241 +0000 UTC m=+75.956883222" Sep 11 15:09:53.938204 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:53.938136 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56f749c75b-dscc4" podStartSLOduration=1.924929675 podStartE2EDuration="5.938116056s" totalImagesPullingTime="4.013186381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:53.937597279 +0000 UTC m=+75.982922260" watchObservedRunningTime="2026-09-11 15:09:53.938116056 +0000 UTC m=+75.983441101" Sep 11 15:09:54.172154 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:54.172124 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm"] Sep 11 15:09:54.174585 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:09:54.174559 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466e17b5_124c_4a5d_8747_98696ecc7701.slice/crio-4b6fc405911d0f5dd7fedf6733f184673d52cf91e471e71307d871f41b872b90 WatchSource:0}: Error finding container 4b6fc405911d0f5dd7fedf6733f184673d52cf91e471e71307d871f41b872b90: Status 404 returned error can't find the container with id 4b6fc405911d0f5dd7fedf6733f184673d52cf91e471e71307d871f41b872b90 Sep 11 15:09:54.898980 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:54.898935 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" event={"ID":"466e17b5-124c-4a5d-8747-98696ecc7701","Type":"ContainerStarted","Data":"4b6fc405911d0f5dd7fedf6733f184673d52cf91e471e71307d871f41b872b90"} Sep 11 15:09:56.907337 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:56.907297 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" event={"ID":"466e17b5-124c-4a5d-8747-98696ecc7701","Type":"ContainerStarted","Data":"1f7670688436284d331c0a83d7f35717952a4db8280e24b282c460053c5106d8"} Sep 11 15:09:56.924652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:56.924602 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-nbpfm" podStartSLOduration=34.225357677 podStartE2EDuration="35.924583146s" totalImagesPullingTime="1.699225469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:56.924088548 +0000 UTC m=+78.969413528" watchObservedRunningTime="2026-09-11 15:09:56.924583146 +0000 UTC m=+78.969908127" Sep 11 15:09:58.815834 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:09:58.815806 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:10:03.610369 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.610242 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fd9db7c5b-brpz9"] Sep 11 15:10:03.613495 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.613474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.618027 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.617997 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 15:10:03.618171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.618151 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 15:10:03.619068 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.619042 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 15:10:03.619173 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.619140 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 15:10:03.619173 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.619140 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wnkdc\"" Sep 11 15:10:03.619289 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.619189 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 15:10:03.619651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.619313 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 15:10:03.619651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.619291 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 15:10:03.623064 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.623048 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 15:10:03.628062 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.628037 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd9db7c5b-brpz9"] Sep 11 15:10:03.697282 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.697255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-trusted-ca-bundle\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.697385 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.697301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-service-ca\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.697385 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.697327 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-serving-cert\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.697385 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.697360 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-config\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.697540 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.697384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q922t\" (UniqueName: \"kubernetes.io/projected/a1554405-2e32-47e6-95d5-2ccfabeaf476-kube-api-access-q922t\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.697540 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.697465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-oauth-serving-cert\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.697540 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.697500 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-oauth-config\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.798569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.798543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-config\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.798661 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.798574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q922t\" (UniqueName: \"kubernetes.io/projected/a1554405-2e32-47e6-95d5-2ccfabeaf476-kube-api-access-q922t\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.798661 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.798614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-oauth-serving-cert\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.798661 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.798647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-oauth-config\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.798914 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.798679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-trusted-ca-bundle\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.798914 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.798723 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-service-ca\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.798914 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.798748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-serving-cert\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.799421 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.799373 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-config\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.799421 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.799373 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-oauth-serving-cert\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.799549 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.799462 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-trusted-ca-bundle\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.799633 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.799613 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-service-ca\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.801249 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.801220 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-oauth-config\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.801249 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.801238 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-serving-cert\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.808123 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.808094 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q922t\" (UniqueName: \"kubernetes.io/projected/a1554405-2e32-47e6-95d5-2ccfabeaf476-kube-api-access-q922t\") pod \"console-fd9db7c5b-brpz9\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:03.900741 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.900694 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p94fh" Sep 11 15:10:03.922509 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:03.922484 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:04.198741 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:04.198678 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd9db7c5b-brpz9"] Sep 11 15:10:04.202085 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:10:04.202061 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1554405_2e32_47e6_95d5_2ccfabeaf476.slice/crio-1a659ab12ecc52b41859bdb86ca29701a53bd31307299ae223c5ed8ae55c66ce WatchSource:0}: Error finding container 1a659ab12ecc52b41859bdb86ca29701a53bd31307299ae223c5ed8ae55c66ce: Status 404 returned error can't find the container with id 1a659ab12ecc52b41859bdb86ca29701a53bd31307299ae223c5ed8ae55c66ce Sep 11 15:10:04.938155 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:04.938122 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd9db7c5b-brpz9" event={"ID":"a1554405-2e32-47e6-95d5-2ccfabeaf476","Type":"ContainerStarted","Data":"1a659ab12ecc52b41859bdb86ca29701a53bd31307299ae223c5ed8ae55c66ce"} Sep 11 15:10:07.950590 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:07.950554 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd9db7c5b-brpz9" event={"ID":"a1554405-2e32-47e6-95d5-2ccfabeaf476","Type":"ContainerStarted","Data":"1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749"} Sep 11 15:10:07.980006 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:07.979959 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fd9db7c5b-brpz9" podStartSLOduration=1.750063721 podStartE2EDuration="4.979943678s" totalImagesPullingTime="3.229879957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:07.979218633 +0000 UTC m=+90.024543613" watchObservedRunningTime="2026-09-11 15:10:07.979943678 +0000 UTC m=+90.025268658" Sep 11 15:10:10.884543 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:10.884515 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78d65bbf48-lstr8" Sep 11 15:10:12.908768 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.908735 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp"] Sep 11 15:10:12.913220 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.913200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:12.915744 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.915722 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-79m44\"" Sep 11 15:10:12.916288 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.916268 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 11 15:10:12.918363 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.916403 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 11 15:10:12.918363 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.916888 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 15:10:12.932240 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.932215 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp"] Sep 11 15:10:12.944603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.944577 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-588rj"] Sep 11 15:10:12.948071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.948055 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mpzrh"] Sep 11 15:10:12.948186 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.948172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:12.951088 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.951070 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:12.954625 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.954594 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 15:10:12.955540 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.955521 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 15:10:12.955637 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.955553 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 15:10:12.956278 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.956255 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 15:10:12.956278 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.956270 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cpl4t\"" Sep 11 15:10:12.956421 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.956266 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 15:10:12.956480 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.956468 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zcpm2\"" Sep 11 15:10:12.956580 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.956559 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 15:10:12.962619 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.962601 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:12.962691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.962632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:12.962691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.962650 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:12.962691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.962667 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbhhd\" (UniqueName: \"kubernetes.io/projected/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-kube-api-access-dbhhd\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:12.981449 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:12.981422 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-588rj"] Sep 11 15:10:13.063827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.063806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82f879dc-186a-4ca5-89e1-31750bdf3e2b-root\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.063924 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.063834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82f879dc-186a-4ca5-89e1-31750bdf3e2b-sys\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.063924 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.063850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-tls\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.063924 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.063871 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:13.063924 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.063891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.064121 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.063923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkcm2\" (UniqueName: \"kubernetes.io/projected/82f879dc-186a-4ca5-89e1-31750bdf3e2b-kube-api-access-xkcm2\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.064121 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.063984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.064121 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.064121 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:13.064121 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82f879dc-186a-4ca5-89e1-31750bdf3e2b-metrics-client-ca\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.064298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-textfile\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.064298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:13.064298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbhhd\" (UniqueName: \"kubernetes.io/projected/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-kube-api-access-dbhhd\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:13.064298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064225 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.064298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064294 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.064617 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064329 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.064617 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.064617 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-wtmp\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.064617 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgh6f\" (UniqueName: \"kubernetes.io/projected/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-api-access-wgh6f\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.064935 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.064739 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:13.066387 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.066368 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:13.066597 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.066580 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:13.075871 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.075842 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbhhd\" (UniqueName: \"kubernetes.io/projected/07e3e6e2-57a7-4f51-8859-25519d5dfb4a-kube-api-access-dbhhd\") pod \"openshift-state-metrics-67dcf8c979-xktpp\" (UID: \"07e3e6e2-57a7-4f51-8859-25519d5dfb4a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:13.165006 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.164946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.165006 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.164990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.165181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.165181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.165181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-wtmp\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.165181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgh6f\" (UniqueName: \"kubernetes.io/projected/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-api-access-wgh6f\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.165181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82f879dc-186a-4ca5-89e1-31750bdf3e2b-root\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.165181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165170 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82f879dc-186a-4ca5-89e1-31750bdf3e2b-sys\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-tls\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165218 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82f879dc-186a-4ca5-89e1-31750bdf3e2b-root\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkcm2\" (UniqueName: \"kubernetes.io/projected/82f879dc-186a-4ca5-89e1-31750bdf3e2b-kube-api-access-xkcm2\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165283 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-wtmp\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165302 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:10:13.165313 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82f879dc-186a-4ca5-89e1-31750bdf3e2b-metrics-client-ca\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:10:13.165388 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-tls podName:82f879dc-186a-4ca5-89e1-31750bdf3e2b nodeName:}" failed. No retries permitted until 2026-09-11 15:10:13.66536446 +0000 UTC m=+95.710689417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-tls") pod "node-exporter-mpzrh" (UID: "82f879dc-186a-4ca5-89e1-31750bdf3e2b") : secret "node-exporter-tls" not found Sep 11 15:10:13.165461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-textfile\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.166038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165468 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82f879dc-186a-4ca5-89e1-31750bdf3e2b-sys\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.166038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165525 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.166038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165687 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-textfile\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.166038 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:10:13.165765 2573 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 11 15:10:13.166038 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:10:13.165847 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-tls podName:e7bd37f1-5c38-4f3f-a055-f1d386d22ac9 nodeName:}" failed. No retries permitted until 2026-09-11 15:10:13.665828906 +0000 UTC m=+95.711153865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-588rj" (UID: "e7bd37f1-5c38-4f3f-a055-f1d386d22ac9") : secret "kube-state-metrics-tls" not found Sep 11 15:10:13.166038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.165997 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.166342 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.166240 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.166395 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.166369 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.166438 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.166396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82f879dc-186a-4ca5-89e1-31750bdf3e2b-metrics-client-ca\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.167525 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.167507 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.167615 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.167583 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.174999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.174976 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgh6f\" (UniqueName: \"kubernetes.io/projected/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-api-access-wgh6f\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.175278 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.175254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkcm2\" (UniqueName: \"kubernetes.io/projected/82f879dc-186a-4ca5-89e1-31750bdf3e2b-kube-api-access-xkcm2\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.226213 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.226182 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" Sep 11 15:10:13.503833 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.503747 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp"] Sep 11 15:10:13.506644 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:10:13.506619 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e3e6e2_57a7_4f51_8859_25519d5dfb4a.slice/crio-634f33520b3f11f5d666c0f28f1a6128e44ad7f9648fca7ae76c2a79cb43007b WatchSource:0}: Error finding container 634f33520b3f11f5d666c0f28f1a6128e44ad7f9648fca7ae76c2a79cb43007b: Status 404 returned error can't find the container with id 634f33520b3f11f5d666c0f28f1a6128e44ad7f9648fca7ae76c2a79cb43007b Sep 11 15:10:13.669973 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.669945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-tls\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:13.670092 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.669986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.670130 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:10:13.670085 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 15:10:13.670167 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:10:13.670146 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-tls podName:82f879dc-186a-4ca5-89e1-31750bdf3e2b nodeName:}" failed. No retries permitted until 2026-09-11 15:10:14.670127917 +0000 UTC m=+96.715452874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-tls") pod "node-exporter-mpzrh" (UID: "82f879dc-186a-4ca5-89e1-31750bdf3e2b") : secret "node-exporter-tls" not found Sep 11 15:10:13.672356 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.672334 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7bd37f1-5c38-4f3f-a055-f1d386d22ac9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-588rj\" (UID: \"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.839808 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.839733 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" podUID="8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" containerName="registry" containerID="cri-o://fd234a17c8e5bbf34acc3707ab44789cb2559056791cd5841cc229550049151f" gracePeriod=30 Sep 11 15:10:13.858629 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.858608 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" Sep 11 15:10:13.923046 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.922997 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:13.923340 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.923083 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:13.928198 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.928177 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:13.974359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.974250 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" event={"ID":"07e3e6e2-57a7-4f51-8859-25519d5dfb4a","Type":"ContainerStarted","Data":"ce05d3f590674a39b759b719ea8daed930d3e9bfa4db9e6933bec3309758c705"} Sep 11 15:10:13.974359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.974306 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" event={"ID":"07e3e6e2-57a7-4f51-8859-25519d5dfb4a","Type":"ContainerStarted","Data":"9ed73020af2fdf864b3969a281b51ae5e06ada87e8ae09068e348091c63bcc10"} Sep 11 15:10:13.974359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.974330 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" event={"ID":"07e3e6e2-57a7-4f51-8859-25519d5dfb4a","Type":"ContainerStarted","Data":"634f33520b3f11f5d666c0f28f1a6128e44ad7f9648fca7ae76c2a79cb43007b"} Sep 11 15:10:13.975600 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.975577 2573 generic.go:356] "Generic (PLEG): container finished" podID="8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" containerID="fd234a17c8e5bbf34acc3707ab44789cb2559056791cd5841cc229550049151f" exitCode=0 Sep 11 15:10:13.976005 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.975979 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" event={"ID":"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c","Type":"ContainerDied","Data":"fd234a17c8e5bbf34acc3707ab44789cb2559056791cd5841cc229550049151f"} Sep 11 15:10:13.980864 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:13.980840 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:10:14.093960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.093913 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:10:14.133060 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.133035 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-588rj"] Sep 11 15:10:14.137469 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:10:14.137430 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bd37f1_5c38_4f3f_a055_f1d386d22ac9.slice/crio-bd9ed5b29196b136b343b070eb17ee875e0f83b7d932177484df170c69b69fcd WatchSource:0}: Error finding container bd9ed5b29196b136b343b070eb17ee875e0f83b7d932177484df170c69b69fcd: Status 404 returned error can't find the container with id bd9ed5b29196b136b343b070eb17ee875e0f83b7d932177484df170c69b69fcd Sep 11 15:10:14.173786 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.173747 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls\") pod \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " Sep 11 15:10:14.173899 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.173837 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-bound-sa-token\") pod \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " Sep 11 15:10:14.173899 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.173871 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-trusted-ca\") pod \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " Sep 11 15:10:14.174012 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.173911 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-certificates\") pod \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " Sep 11 15:10:14.174012 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.173946 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-image-registry-private-configuration\") pod \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " Sep 11 15:10:14.174012 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.173984 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-ca-trust-extracted\") pod \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " Sep 11 15:10:14.174153 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.174015 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-installation-pull-secrets\") pod \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " Sep 11 15:10:14.175112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.174717 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9vvl\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-kube-api-access-p9vvl\") pod \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\" (UID: \"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c\") " Sep 11 15:10:14.175112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.174871 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:14.175112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.175075 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-trusted-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:14.175538 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.175504 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:14.178159 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.178103 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:10:14.178398 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.178357 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-kube-api-access-p9vvl" (OuterVolumeSpecName: "kube-api-access-p9vvl") pod "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c"). InnerVolumeSpecName "kube-api-access-p9vvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:10:14.178602 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.178574 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:10:14.179161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.178915 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:10:14.179161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.179044 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:10:14.186344 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.186320 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" (UID: "8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:10:14.275744 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.275714 2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-bound-sa-token\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:14.275744 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.275746 2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-certificates\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:14.275960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.275762 2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-image-registry-private-configuration\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:14.275960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.275793 2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-ca-trust-extracted\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:14.275960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.275807 2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-installation-pull-secrets\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:14.275960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.275821 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9vvl\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-kube-api-access-p9vvl\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:14.275960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.275836 2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c-registry-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:14.679463 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.679434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-tls\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:14.681699 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.681670 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82f879dc-186a-4ca5-89e1-31750bdf3e2b-node-exporter-tls\") pod \"node-exporter-mpzrh\" (UID: \"82f879dc-186a-4ca5-89e1-31750bdf3e2b\") " pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:14.762510 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.762489 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpzrh" Sep 11 15:10:14.785739 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:10:14.785717 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f879dc_186a_4ca5_89e1_31750bdf3e2b.slice/crio-8191fc524176171d2b00517c588a11849f603c3a6887017eb4eac8fa3e520fd7 WatchSource:0}: Error finding container 8191fc524176171d2b00517c588a11849f603c3a6887017eb4eac8fa3e520fd7: Status 404 returned error can't find the container with id 8191fc524176171d2b00517c588a11849f603c3a6887017eb4eac8fa3e520fd7 Sep 11 15:10:14.867937 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.867906 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-54cb8c4954-rsjln"] Sep 11 15:10:14.868277 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.868260 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" containerName="registry" Sep 11 15:10:14.868353 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.868280 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" containerName="registry" Sep 11 15:10:14.868353 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.868325 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" containerName="registry" Sep 11 15:10:14.871583 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.871555 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:14.876758 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.876741 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 11 15:10:14.877185 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.877170 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 11 15:10:14.877850 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.877811 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 11 15:10:14.878015 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.877995 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8557l\"" Sep 11 15:10:14.878128 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.878108 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9vd9e7mt0fslu\"" Sep 11 15:10:14.879011 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.878830 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 11 15:10:14.880232 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.880200 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 11 15:10:14.909245 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.909220 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54cb8c4954-rsjln"] Sep 11 15:10:14.981175 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.981142 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:14.981569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.981204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:14.981569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.981250 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg4kk\" (UniqueName: \"kubernetes.io/projected/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-kube-api-access-jg4kk\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:14.981569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.981311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-grpc-tls\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:14.981569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.981354 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-tls\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:14.981569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.981386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:14.981569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.981421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-metrics-client-ca\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:14.981569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.981459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:14.982161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.982125 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" event={"ID":"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9","Type":"ContainerStarted","Data":"bd9ed5b29196b136b343b070eb17ee875e0f83b7d932177484df170c69b69fcd"} Sep 11 15:10:14.984699 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.984638 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" event={"ID":"07e3e6e2-57a7-4f51-8859-25519d5dfb4a","Type":"ContainerStarted","Data":"e25d28c6321161637ebfb23992095019cd5769cd60c1f503cf2266e3e0b159d8"} Sep 11 15:10:14.986195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.986172 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpzrh" event={"ID":"82f879dc-186a-4ca5-89e1-31750bdf3e2b","Type":"ContainerStarted","Data":"8191fc524176171d2b00517c588a11849f603c3a6887017eb4eac8fa3e520fd7"} Sep 11 15:10:14.987455 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.987438 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" Sep 11 15:10:14.987541 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.987477 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8f6b79cd-5c7gp" event={"ID":"8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c","Type":"ContainerDied","Data":"2b6a2a8d680248aff97c331038b404c9b23b1a8eed298c634d89b0a2dfca8ed5"} Sep 11 15:10:14.987541 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:14.987523 2573 scope.go:117] "RemoveContainer" containerID="fd234a17c8e5bbf34acc3707ab44789cb2559056791cd5841cc229550049151f" Sep 11 15:10:15.006134 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.006097 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xktpp" podStartSLOduration=1.892660789 podStartE2EDuration="3.006084478s" totalImagesPullingTime="1.113423689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:15.005055293 +0000 UTC m=+97.050380274" watchObservedRunningTime="2026-09-11 15:10:15.006084478 +0000 UTC m=+97.051409457" Sep 11 15:10:15.020105 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.020058 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c8f6b79cd-5c7gp"] Sep 11 15:10:15.024260 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.024243 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c8f6b79cd-5c7gp"] Sep 11 15:10:15.082439 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.082410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.082542 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.082497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.082542 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.082524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4kk\" (UniqueName: \"kubernetes.io/projected/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-kube-api-access-jg4kk\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.082665 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.082551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-grpc-tls\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.082665 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.082584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-tls\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.082665 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.082631 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.082853 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.082696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-metrics-client-ca\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.082853 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.082760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.083484 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.083458 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-metrics-client-ca\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.085583 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.085560 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.085800 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.085755 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-tls\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.085935 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.085914 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.086309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.086275 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.086416 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.086368 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.086857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.086840 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-secret-grpc-tls\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.092819 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.092797 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg4kk\" (UniqueName: \"kubernetes.io/projected/8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c-kube-api-access-jg4kk\") pod \"thanos-querier-54cb8c4954-rsjln\" (UID: \"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c\") " pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.183650 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.183628 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:15.465038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.464985 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54cb8c4954-rsjln"] Sep 11 15:10:15.639657 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:10:15.639628 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6c127f_ac4d_40f7_a70c_1e2d947b0b7c.slice/crio-c96cc268f5d94ddd047b46fc19a53914ec556bec10cdfe9b60592a96c301df1d WatchSource:0}: Error finding container c96cc268f5d94ddd047b46fc19a53914ec556bec10cdfe9b60592a96c301df1d: Status 404 returned error can't find the container with id c96cc268f5d94ddd047b46fc19a53914ec556bec10cdfe9b60592a96c301df1d Sep 11 15:10:15.996071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.995980 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" event={"ID":"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9","Type":"ContainerStarted","Data":"3fcdf15af67b93d07cee869f5644bb85e783ade1501d67b6ea8f447ce72e57e5"} Sep 11 15:10:15.996071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.996055 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" event={"ID":"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9","Type":"ContainerStarted","Data":"7fc78edffcb5ce2f5f12c6ff2c2991aeb50301301e44db935f76e544e3bb53f0"} Sep 11 15:10:15.996568 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.996084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" event={"ID":"e7bd37f1-5c38-4f3f-a055-f1d386d22ac9","Type":"ContainerStarted","Data":"6b9839f217dc93fce856f090b4ac21156f75b226d3030cc9211eb2412212ff5a"} Sep 11 15:10:15.997859 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:15.997830 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" event={"ID":"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c","Type":"ContainerStarted","Data":"c96cc268f5d94ddd047b46fc19a53914ec556bec10cdfe9b60592a96c301df1d"} Sep 11 15:10:16.015139 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:16.015077 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-588rj" podStartSLOduration=2.471419783 podStartE2EDuration="4.015057094s" totalImagesPullingTime="1.543637311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:16.014163336 +0000 UTC m=+98.059488317" watchObservedRunningTime="2026-09-11 15:10:16.015057094 +0000 UTC m=+98.060382073" Sep 11 15:10:16.494538 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:16.494158 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c" path="/var/lib/kubelet/pods/8f88c4ef-6848-4c12-8f42-c4bdb2d9a46c/volumes" Sep 11 15:10:17.004154 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.004119 2573 generic.go:356] "Generic (PLEG): container finished" podID="82f879dc-186a-4ca5-89e1-31750bdf3e2b" containerID="e3a19b665436b4eb3fa5086b3781678ec09044e168eab091e716d1eca531622d" exitCode=0 Sep 11 15:10:17.004582 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.004217 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpzrh" event={"ID":"82f879dc-186a-4ca5-89e1-31750bdf3e2b","Type":"ContainerDied","Data":"e3a19b665436b4eb3fa5086b3781678ec09044e168eab091e716d1eca531622d"} Sep 11 15:10:17.587265 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.587228 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd"] Sep 11 15:10:17.590180 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.590158 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" Sep 11 15:10:17.593029 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.592456 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bc8kb\"" Sep 11 15:10:17.593029 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.592726 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 15:10:17.603992 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.603972 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd"] Sep 11 15:10:17.705455 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.705403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxh5\" (UniqueName: \"kubernetes.io/projected/9929bde3-5ee9-4f05-8e98-7a871c7355e0-kube-api-access-6jxh5\") pod \"monitoring-plugin-5d85778b9d-ttlfd\" (UID: \"9929bde3-5ee9-4f05-8e98-7a871c7355e0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" Sep 11 15:10:17.705455 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.705437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9929bde3-5ee9-4f05-8e98-7a871c7355e0-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-ttlfd\" (UID: \"9929bde3-5ee9-4f05-8e98-7a871c7355e0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" Sep 11 15:10:17.800817 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.800792 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dbdf468b5-7rqmf"] Sep 11 15:10:17.806672 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.806648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxh5\" (UniqueName: \"kubernetes.io/projected/9929bde3-5ee9-4f05-8e98-7a871c7355e0-kube-api-access-6jxh5\") pod \"monitoring-plugin-5d85778b9d-ttlfd\" (UID: \"9929bde3-5ee9-4f05-8e98-7a871c7355e0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" Sep 11 15:10:17.806785 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.806687 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9929bde3-5ee9-4f05-8e98-7a871c7355e0-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-ttlfd\" (UID: \"9929bde3-5ee9-4f05-8e98-7a871c7355e0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" Sep 11 15:10:17.809190 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.809169 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9929bde3-5ee9-4f05-8e98-7a871c7355e0-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-ttlfd\" (UID: \"9929bde3-5ee9-4f05-8e98-7a871c7355e0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" Sep 11 15:10:17.812462 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.812445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:17.817195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.817175 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dbdf468b5-7rqmf"] Sep 11 15:10:17.820560 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.820529 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxh5\" (UniqueName: \"kubernetes.io/projected/9929bde3-5ee9-4f05-8e98-7a871c7355e0-kube-api-access-6jxh5\") pod \"monitoring-plugin-5d85778b9d-ttlfd\" (UID: \"9929bde3-5ee9-4f05-8e98-7a871c7355e0\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" Sep 11 15:10:17.901202 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.901176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" Sep 11 15:10:17.907817 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.907791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-serving-cert\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:17.907881 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.907842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-trusted-ca-bundle\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:17.907881 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.907871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdbtx\" (UniqueName: \"kubernetes.io/projected/c6edb816-b6c2-4eb8-906f-250092271b4f-kube-api-access-gdbtx\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:17.907940 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.907889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-oauth-serving-cert\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:17.907940 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.907908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-console-config\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:17.907999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.907945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-oauth-config\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:17.907999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:17.907968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-service-ca\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.008358 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.008327 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-trusted-ca-bundle\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.008793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.008376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdbtx\" (UniqueName: \"kubernetes.io/projected/c6edb816-b6c2-4eb8-906f-250092271b4f-kube-api-access-gdbtx\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.008793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.008511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-oauth-serving-cert\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.008793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.008555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-console-config\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.008793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.008600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-oauth-config\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.008793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.008622 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-service-ca\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.008793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.008688 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-serving-cert\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.009378 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.009267 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-oauth-serving-cert\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.009378 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.009316 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-trusted-ca-bundle\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.009500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.009486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-console-config\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.009537 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.009508 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-service-ca\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.010517 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.010490 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpzrh" event={"ID":"82f879dc-186a-4ca5-89e1-31750bdf3e2b","Type":"ContainerStarted","Data":"38f142ff89bbdc194f7a1f3db8edd8bfb1573e45277f8b0bc561ae16d622628e"} Sep 11 15:10:18.010644 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.010548 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpzrh" event={"ID":"82f879dc-186a-4ca5-89e1-31750bdf3e2b","Type":"ContainerStarted","Data":"43c501f466c2edf97eb9af5daa924f79669b6800001faadd12c1943e779065e1"} Sep 11 15:10:18.011640 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.011613 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-oauth-config\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.011822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.011799 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-serving-cert\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.012953 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.012926 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" event={"ID":"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c","Type":"ContainerStarted","Data":"36c07d48446d3b61c6bd443b30a039e61535a3914210d61c79e490cafd7a80f7"} Sep 11 15:10:18.013017 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.012994 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" event={"ID":"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c","Type":"ContainerStarted","Data":"55cf74e9d3fe6f29e0b180a8638d5e1063ca39dd0eaf53bff0434e10f0b56645"} Sep 11 15:10:18.013071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.013020 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" event={"ID":"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c","Type":"ContainerStarted","Data":"171d0505a8863bd9ff74a151d0327843aadb92c7d8e2c1f8caf9391b0fbcf5bf"} Sep 11 15:10:18.017947 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.017931 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdbtx\" (UniqueName: \"kubernetes.io/projected/c6edb816-b6c2-4eb8-906f-250092271b4f-kube-api-access-gdbtx\") pod \"console-dbdf468b5-7rqmf\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.035787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.035738 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mpzrh" podStartSLOduration=4.367804477 podStartE2EDuration="6.035725747s" totalImagesPullingTime="1.592445655s" totalInitContainerRuntime="75.475615ms" isStatefulPod=true podCreationTimestamp="2026-09-11 15:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:18.034734506 +0000 UTC m=+100.080059510" watchObservedRunningTime="2026-09-11 15:10:18.035725747 +0000 UTC m=+100.081050782" Sep 11 15:10:18.122204 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.122179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:18.206513 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.206487 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd"] Sep 11 15:10:18.210027 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:10:18.209760 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9929bde3_5ee9_4f05_8e98_7a871c7355e0.slice/crio-41c43b6d4db1e7f64744dfc795d819a555926602ada3142876d702da08ca787c WatchSource:0}: Error finding container 41c43b6d4db1e7f64744dfc795d819a555926602ada3142876d702da08ca787c: Status 404 returned error can't find the container with id 41c43b6d4db1e7f64744dfc795d819a555926602ada3142876d702da08ca787c Sep 11 15:10:18.425622 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:18.425593 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dbdf468b5-7rqmf"] Sep 11 15:10:18.428568 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:10:18.428529 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6edb816_b6c2_4eb8_906f_250092271b4f.slice/crio-e83b3b3c9ca4d388cf8d63369f0c77a8630ea697dd3d1af53c4b79cb00444efa WatchSource:0}: Error finding container e83b3b3c9ca4d388cf8d63369f0c77a8630ea697dd3d1af53c4b79cb00444efa: Status 404 returned error can't find the container with id e83b3b3c9ca4d388cf8d63369f0c77a8630ea697dd3d1af53c4b79cb00444efa Sep 11 15:10:19.019267 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.019230 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" event={"ID":"9929bde3-5ee9-4f05-8e98-7a871c7355e0","Type":"ContainerStarted","Data":"41c43b6d4db1e7f64744dfc795d819a555926602ada3142876d702da08ca787c"} Sep 11 15:10:19.022080 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.021967 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dbdf468b5-7rqmf" event={"ID":"c6edb816-b6c2-4eb8-906f-250092271b4f","Type":"ContainerStarted","Data":"7af638c4c9ab9d1accb53efadb5ed9d9578ebf2f54c43debe38669e8975ddfc2"} Sep 11 15:10:19.022080 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.022012 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dbdf468b5-7rqmf" event={"ID":"c6edb816-b6c2-4eb8-906f-250092271b4f","Type":"ContainerStarted","Data":"e83b3b3c9ca4d388cf8d63369f0c77a8630ea697dd3d1af53c4b79cb00444efa"} Sep 11 15:10:19.045479 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.045429 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dbdf468b5-7rqmf" podStartSLOduration=2.045410968 podStartE2EDuration="2.045410968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:19.043943669 +0000 UTC m=+101.089268650" watchObservedRunningTime="2026-09-11 15:10:19.045410968 +0000 UTC m=+101.090735948" Sep 11 15:10:19.127485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.127463 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 15:10:19.143125 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.143106 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.145986 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.145962 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 15:10:19.146068 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.145991 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 15:10:19.146068 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146003 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 15:10:19.146250 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146222 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 15:10:19.146311 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146292 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 15:10:19.146368 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146337 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 15:10:19.146368 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146346 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 15:10:19.146469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146379 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 15:10:19.146469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146390 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 15:10:19.146566 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146383 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 15:10:19.146566 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146513 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 15:10:19.146668 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146577 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 15:10:19.146668 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146577 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 15:10:19.146828 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146808 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9prk4\"" Sep 11 15:10:19.146881 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146846 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dd9sqqrbimsvq\"" Sep 11 15:10:19.146955 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.146943 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 15:10:19.149830 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.149807 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 15:10:19.150659 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.150638 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 15:10:19.218509 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.218490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.218591 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.218520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.218591 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.218539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.218591 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.218565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-web-config\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.218721 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.218649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.218721 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.218689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.218923 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.218907 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.218968 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.218942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-config-out\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219014 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.218999 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219046 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219086 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219186 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219186 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219161 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219247 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219280 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219312 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr48d\" (UniqueName: \"kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-kube-api-access-fr48d\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219375 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219340 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.219375 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.219366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-config\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320431 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320531 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320439 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320531 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320464 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320531 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320531 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320496 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320531 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320583 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr48d\" (UniqueName: \"kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-kube-api-access-fr48d\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-config\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.320787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.321224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-web-config\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.321224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320838 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.321224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.321224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.321224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.321224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.320964 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-config-out\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.321517 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.321252 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.321517 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.321252 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.321641 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.321615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.322562 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.322538 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.323357 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.323332 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.323885 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.323669 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-config-out\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.323885 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.323740 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.324097 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.324039 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.324196 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.324131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.324892 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.324872 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.325340 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.325302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.325340 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.325328 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.326674 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.326126 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.326674 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.326195 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-config\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.326674 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.326324 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.326674 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.326502 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.327151 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.327129 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.327594 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.327573 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-web-config\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.329030 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.329009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr48d\" (UniqueName: \"kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-kube-api-access-fr48d\") pod \"prometheus-k8s-0\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.454294 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.454273 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:19.770334 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:19.770307 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 15:10:19.772563 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:10:19.772535 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974c5423_97e4_4180_8beb_741511108f83.slice/crio-9007456928cbea63ccf19fb223df09f3996acc144fd9463afec1b21e5dfc7a11 WatchSource:0}: Error finding container 9007456928cbea63ccf19fb223df09f3996acc144fd9463afec1b21e5dfc7a11: Status 404 returned error can't find the container with id 9007456928cbea63ccf19fb223df09f3996acc144fd9463afec1b21e5dfc7a11 Sep 11 15:10:20.028685 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:20.028651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" event={"ID":"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c","Type":"ContainerStarted","Data":"4fb0f9918d2f4e242f4f29b93deb7be81254306ba832ce0636b140e6050f91fa"} Sep 11 15:10:20.029194 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:20.028739 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" event={"ID":"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c","Type":"ContainerStarted","Data":"3534d97552207482da28800ddd468f8bb3af2b753aad9358c049816eff8e0766"} Sep 11 15:10:20.029194 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:20.028803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" event={"ID":"8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c","Type":"ContainerStarted","Data":"fbe01a2cd412d0718ac687a92c53e7c5b1af1a425cdafcf16991ed5494f31cf5"} Sep 11 15:10:20.029194 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:20.028849 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:20.030026 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:20.030003 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerStarted","Data":"9007456928cbea63ccf19fb223df09f3996acc144fd9463afec1b21e5dfc7a11"} Sep 11 15:10:20.048950 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:20.048902 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" podStartSLOduration=2.798286536 podStartE2EDuration="6.048887697s" totalImagesPullingTime="3.250601161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:20.048072839 +0000 UTC m=+102.093397819" watchObservedRunningTime="2026-09-11 15:10:20.048887697 +0000 UTC m=+102.094212711" Sep 11 15:10:20.882614 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:20.882591 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7z2zb" Sep 11 15:10:21.036351 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:21.036312 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" event={"ID":"9929bde3-5ee9-4f05-8e98-7a871c7355e0","Type":"ContainerStarted","Data":"c33014c70d8f5055f8eea27acf5d086b1cedad340d9d59e2b8a2b05e67ab6134"} Sep 11 15:10:21.036820 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:21.036803 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" Sep 11 15:10:21.042026 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:21.041998 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" Sep 11 15:10:21.051893 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:21.051851 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-ttlfd" podStartSLOduration=1.433252456 podStartE2EDuration="4.051831584s" totalImagesPullingTime="2.618579128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:21.051574169 +0000 UTC m=+103.096899150" watchObservedRunningTime="2026-09-11 15:10:21.051831584 +0000 UTC m=+103.097156569" Sep 11 15:10:22.041817 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:22.041740 2573 generic.go:356] "Generic (PLEG): container finished" podID="974c5423-97e4-4180-8beb-741511108f83" containerID="9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37" exitCode=0 Sep 11 15:10:22.042113 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:22.041825 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerDied","Data":"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37"} Sep 11 15:10:23.912635 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:23.912599 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dbdf468b5-7rqmf"] Sep 11 15:10:23.984429 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:23.984399 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d7bdc59-2qqkv"] Sep 11 15:10:24.000923 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.000903 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.029486 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.029451 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d7bdc59-2qqkv"] Sep 11 15:10:24.064830 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.064798 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-oauth-serving-cert\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.065001 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.064856 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-trusted-ca-bundle\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.065001 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.064897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-serving-cert\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.065001 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.064966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mrh\" (UniqueName: \"kubernetes.io/projected/c5e68cba-91fa-45a1-a639-034e9f9af6ef-kube-api-access-w7mrh\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.065155 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.065074 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-config\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.065155 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.065115 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-service-ca\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.065245 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.065166 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-oauth-config\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.165872 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.165791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mrh\" (UniqueName: \"kubernetes.io/projected/c5e68cba-91fa-45a1-a639-034e9f9af6ef-kube-api-access-w7mrh\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.166018 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.165878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-config\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.166018 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.165920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-service-ca\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.166178 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.166096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-oauth-config\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.166178 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.166161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-oauth-serving-cert\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.166298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.166197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-trusted-ca-bundle\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.166298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.166216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-serving-cert\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.166666 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.166642 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-config\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.166800 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.166701 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-service-ca\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.167386 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.167359 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-oauth-serving-cert\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.167468 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.167383 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-trusted-ca-bundle\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.169075 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.169056 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-serving-cert\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.169189 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.169171 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-oauth-config\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.177194 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.177169 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mrh\" (UniqueName: \"kubernetes.io/projected/c5e68cba-91fa-45a1-a639-034e9f9af6ef-kube-api-access-w7mrh\") pod \"console-d7bdc59-2qqkv\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:24.310347 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:24.310320 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:25.400845 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:25.400816 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d7bdc59-2qqkv"] Sep 11 15:10:25.448673 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:10:25.448627 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e68cba_91fa_45a1_a639_034e9f9af6ef.slice/crio-03f616e3678fbd8efb48332f14f12abf0411db16b0705c70f2c29208f6d4e352 WatchSource:0}: Error finding container 03f616e3678fbd8efb48332f14f12abf0411db16b0705c70f2c29208f6d4e352: Status 404 returned error can't find the container with id 03f616e3678fbd8efb48332f14f12abf0411db16b0705c70f2c29208f6d4e352 Sep 11 15:10:26.042928 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:26.042907 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-54cb8c4954-rsjln" Sep 11 15:10:26.064756 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:26.064725 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerStarted","Data":"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387"} Sep 11 15:10:26.064857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:26.064810 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerStarted","Data":"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8"} Sep 11 15:10:26.064857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:26.064843 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerStarted","Data":"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273"} Sep 11 15:10:26.064958 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:26.064874 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerStarted","Data":"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42"} Sep 11 15:10:26.066434 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:26.066409 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7bdc59-2qqkv" event={"ID":"c5e68cba-91fa-45a1-a639-034e9f9af6ef","Type":"ContainerStarted","Data":"b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519"} Sep 11 15:10:26.066536 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:26.066453 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7bdc59-2qqkv" event={"ID":"c5e68cba-91fa-45a1-a639-034e9f9af6ef","Type":"ContainerStarted","Data":"03f616e3678fbd8efb48332f14f12abf0411db16b0705c70f2c29208f6d4e352"} Sep 11 15:10:26.089950 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:26.089899 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d7bdc59-2qqkv" podStartSLOduration=3.089883426 podStartE2EDuration="3.089883426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:26.088571163 +0000 UTC m=+108.133896145" watchObservedRunningTime="2026-09-11 15:10:26.089883426 +0000 UTC m=+108.135208403" Sep 11 15:10:27.075864 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:27.075832 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerStarted","Data":"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2"} Sep 11 15:10:27.076242 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:27.075894 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerStarted","Data":"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7"} Sep 11 15:10:27.117020 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:27.116956 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.702464436 podStartE2EDuration="8.116933489s" totalImagesPullingTime="5.392179079s" totalInitContainerRuntime="22.289974ms" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:27.112003881 +0000 UTC m=+109.157328863" watchObservedRunningTime="2026-09-11 15:10:27.116933489 +0000 UTC m=+109.162258471" Sep 11 15:10:28.123062 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:28.123032 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:29.454878 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:29.454844 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:10:34.310688 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:34.310648 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:34.311183 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:34.310816 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:34.315306 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:34.315282 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:35.112539 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:35.112509 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:10:35.161192 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:35.161159 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd9db7c5b-brpz9"] Sep 11 15:10:41.134804 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:41.134751 2573 generic.go:356] "Generic (PLEG): container finished" podID="a709ba86-240f-4cdf-afc4-69a224583c98" containerID="52af77789376eb83d85254195d5bfecd4f4f19b94506a5cebd717606bc02b623" exitCode=0 Sep 11 15:10:41.135226 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:41.134827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" event={"ID":"a709ba86-240f-4cdf-afc4-69a224583c98","Type":"ContainerDied","Data":"52af77789376eb83d85254195d5bfecd4f4f19b94506a5cebd717606bc02b623"} Sep 11 15:10:41.135226 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:41.135105 2573 scope.go:117] "RemoveContainer" containerID="52af77789376eb83d85254195d5bfecd4f4f19b94506a5cebd717606bc02b623" Sep 11 15:10:42.141452 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:42.141417 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkr78" event={"ID":"a709ba86-240f-4cdf-afc4-69a224583c98","Type":"ContainerStarted","Data":"cfc953314625cb3b70b3c03160702ce54b073d5348f1db5a98aca630d4499983"} Sep 11 15:10:48.937001 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:48.936963 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dbdf468b5-7rqmf" podUID="c6edb816-b6c2-4eb8-906f-250092271b4f" containerName="console" containerID="cri-o://7af638c4c9ab9d1accb53efadb5ed9d9578ebf2f54c43debe38669e8975ddfc2" gracePeriod=15 Sep 11 15:10:49.172932 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.172896 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6edb816-b6c2-4eb8-906f-250092271b4f" containerID="7af638c4c9ab9d1accb53efadb5ed9d9578ebf2f54c43debe38669e8975ddfc2" exitCode=0 Sep 11 15:10:49.173040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.172966 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dbdf468b5-7rqmf" event={"ID":"c6edb816-b6c2-4eb8-906f-250092271b4f","Type":"ContainerDied","Data":"7af638c4c9ab9d1accb53efadb5ed9d9578ebf2f54c43debe38669e8975ddfc2"} Sep 11 15:10:49.210303 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.210261 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:49.270685 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.270656 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-oauth-serving-cert\") pod \"c6edb816-b6c2-4eb8-906f-250092271b4f\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " Sep 11 15:10:49.270836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.270695 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-trusted-ca-bundle\") pod \"c6edb816-b6c2-4eb8-906f-250092271b4f\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " Sep 11 15:10:49.270836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.270714 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-service-ca\") pod \"c6edb816-b6c2-4eb8-906f-250092271b4f\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " Sep 11 15:10:49.270836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.270728 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-console-config\") pod \"c6edb816-b6c2-4eb8-906f-250092271b4f\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " Sep 11 15:10:49.270836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.270744 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-oauth-config\") pod \"c6edb816-b6c2-4eb8-906f-250092271b4f\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " Sep 11 15:10:49.270836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.270818 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-serving-cert\") pod \"c6edb816-b6c2-4eb8-906f-250092271b4f\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " Sep 11 15:10:49.271094 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.270840 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdbtx\" (UniqueName: \"kubernetes.io/projected/c6edb816-b6c2-4eb8-906f-250092271b4f-kube-api-access-gdbtx\") pod \"c6edb816-b6c2-4eb8-906f-250092271b4f\" (UID: \"c6edb816-b6c2-4eb8-906f-250092271b4f\") " Sep 11 15:10:49.271182 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.271152 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c6edb816-b6c2-4eb8-906f-250092271b4f" (UID: "c6edb816-b6c2-4eb8-906f-250092271b4f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:49.271232 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.271165 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-service-ca" (OuterVolumeSpecName: "service-ca") pod "c6edb816-b6c2-4eb8-906f-250092271b4f" (UID: "c6edb816-b6c2-4eb8-906f-250092271b4f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:49.271232 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.271176 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c6edb816-b6c2-4eb8-906f-250092271b4f" (UID: "c6edb816-b6c2-4eb8-906f-250092271b4f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:49.271232 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.271180 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-console-config" (OuterVolumeSpecName: "console-config") pod "c6edb816-b6c2-4eb8-906f-250092271b4f" (UID: "c6edb816-b6c2-4eb8-906f-250092271b4f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:49.273264 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.273238 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c6edb816-b6c2-4eb8-906f-250092271b4f" (UID: "c6edb816-b6c2-4eb8-906f-250092271b4f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:10:49.273356 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.273330 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c6edb816-b6c2-4eb8-906f-250092271b4f" (UID: "c6edb816-b6c2-4eb8-906f-250092271b4f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:10:49.273393 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.273364 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6edb816-b6c2-4eb8-906f-250092271b4f-kube-api-access-gdbtx" (OuterVolumeSpecName: "kube-api-access-gdbtx") pod "c6edb816-b6c2-4eb8-906f-250092271b4f" (UID: "c6edb816-b6c2-4eb8-906f-250092271b4f"). InnerVolumeSpecName "kube-api-access-gdbtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:10:49.371999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.371975 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-trusted-ca-bundle\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:49.371999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.371995 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:49.372113 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.372005 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-console-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:49.372113 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.372012 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-oauth-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:49.372113 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.372020 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6edb816-b6c2-4eb8-906f-250092271b4f-console-serving-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:49.372113 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.372029 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdbtx\" (UniqueName: \"kubernetes.io/projected/c6edb816-b6c2-4eb8-906f-250092271b4f-kube-api-access-gdbtx\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:49.372113 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:49.372038 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6edb816-b6c2-4eb8-906f-250092271b4f-oauth-serving-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:10:50.177799 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:50.177751 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dbdf468b5-7rqmf" Sep 11 15:10:50.178274 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:50.177756 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dbdf468b5-7rqmf" event={"ID":"c6edb816-b6c2-4eb8-906f-250092271b4f","Type":"ContainerDied","Data":"e83b3b3c9ca4d388cf8d63369f0c77a8630ea697dd3d1af53c4b79cb00444efa"} Sep 11 15:10:50.178274 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:50.177922 2573 scope.go:117] "RemoveContainer" containerID="7af638c4c9ab9d1accb53efadb5ed9d9578ebf2f54c43debe38669e8975ddfc2" Sep 11 15:10:50.198039 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:50.198013 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dbdf468b5-7rqmf"] Sep 11 15:10:50.202001 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:50.201978 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dbdf468b5-7rqmf"] Sep 11 15:10:50.492907 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:10:50.492834 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6edb816-b6c2-4eb8-906f-250092271b4f" path="/var/lib/kubelet/pods/c6edb816-b6c2-4eb8-906f-250092271b4f/volumes" Sep 11 15:11:00.182553 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.182498 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fd9db7c5b-brpz9" podUID="a1554405-2e32-47e6-95d5-2ccfabeaf476" containerName="console" containerID="cri-o://1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749" gracePeriod=15 Sep 11 15:11:00.420200 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.420173 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:11:00.453573 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.453513 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-config\") pod \"a1554405-2e32-47e6-95d5-2ccfabeaf476\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " Sep 11 15:11:00.453573 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.453567 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q922t\" (UniqueName: \"kubernetes.io/projected/a1554405-2e32-47e6-95d5-2ccfabeaf476-kube-api-access-q922t\") pod \"a1554405-2e32-47e6-95d5-2ccfabeaf476\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " Sep 11 15:11:00.453722 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.453591 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-trusted-ca-bundle\") pod \"a1554405-2e32-47e6-95d5-2ccfabeaf476\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " Sep 11 15:11:00.453722 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.453610 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-serving-cert\") pod \"a1554405-2e32-47e6-95d5-2ccfabeaf476\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " Sep 11 15:11:00.453722 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.453627 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-oauth-serving-cert\") pod \"a1554405-2e32-47e6-95d5-2ccfabeaf476\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " Sep 11 15:11:00.453722 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.453676 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-service-ca\") pod \"a1554405-2e32-47e6-95d5-2ccfabeaf476\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " Sep 11 15:11:00.453922 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.453726 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-oauth-config\") pod \"a1554405-2e32-47e6-95d5-2ccfabeaf476\" (UID: \"a1554405-2e32-47e6-95d5-2ccfabeaf476\") " Sep 11 15:11:00.454156 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.454133 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a1554405-2e32-47e6-95d5-2ccfabeaf476" (UID: "a1554405-2e32-47e6-95d5-2ccfabeaf476"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:00.454255 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.454098 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-config" (OuterVolumeSpecName: "console-config") pod "a1554405-2e32-47e6-95d5-2ccfabeaf476" (UID: "a1554405-2e32-47e6-95d5-2ccfabeaf476"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:00.454383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.454359 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-service-ca" (OuterVolumeSpecName: "service-ca") pod "a1554405-2e32-47e6-95d5-2ccfabeaf476" (UID: "a1554405-2e32-47e6-95d5-2ccfabeaf476"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:00.454607 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.454405 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a1554405-2e32-47e6-95d5-2ccfabeaf476" (UID: "a1554405-2e32-47e6-95d5-2ccfabeaf476"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:00.456015 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.455986 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1554405-2e32-47e6-95d5-2ccfabeaf476-kube-api-access-q922t" (OuterVolumeSpecName: "kube-api-access-q922t") pod "a1554405-2e32-47e6-95d5-2ccfabeaf476" (UID: "a1554405-2e32-47e6-95d5-2ccfabeaf476"). InnerVolumeSpecName "kube-api-access-q922t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:11:00.456146 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.456129 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a1554405-2e32-47e6-95d5-2ccfabeaf476" (UID: "a1554405-2e32-47e6-95d5-2ccfabeaf476"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:00.456452 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.456432 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a1554405-2e32-47e6-95d5-2ccfabeaf476" (UID: "a1554405-2e32-47e6-95d5-2ccfabeaf476"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:00.555112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.555081 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-oauth-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:00.555112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.555107 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:00.555298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.555123 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q922t\" (UniqueName: \"kubernetes.io/projected/a1554405-2e32-47e6-95d5-2ccfabeaf476-kube-api-access-q922t\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:00.555298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.555136 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-trusted-ca-bundle\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:00.555298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.555148 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1554405-2e32-47e6-95d5-2ccfabeaf476-console-serving-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:00.555298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.555160 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-oauth-serving-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:00.555298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:00.555171 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1554405-2e32-47e6-95d5-2ccfabeaf476-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:01.225017 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:01.224981 2573 generic.go:356] "Generic (PLEG): container finished" podID="a1554405-2e32-47e6-95d5-2ccfabeaf476" containerID="1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749" exitCode=0 Sep 11 15:11:01.225492 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:01.225037 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd9db7c5b-brpz9" Sep 11 15:11:01.225492 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:01.225067 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd9db7c5b-brpz9" event={"ID":"a1554405-2e32-47e6-95d5-2ccfabeaf476","Type":"ContainerDied","Data":"1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749"} Sep 11 15:11:01.225492 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:01.225129 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd9db7c5b-brpz9" event={"ID":"a1554405-2e32-47e6-95d5-2ccfabeaf476","Type":"ContainerDied","Data":"1a659ab12ecc52b41859bdb86ca29701a53bd31307299ae223c5ed8ae55c66ce"} Sep 11 15:11:01.225492 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:01.225151 2573 scope.go:117] "RemoveContainer" containerID="1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749" Sep 11 15:11:01.234107 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:01.234090 2573 scope.go:117] "RemoveContainer" containerID="1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749" Sep 11 15:11:01.234401 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:11:01.234381 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749\": container with ID starting with 1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749 not found: ID does not exist" containerID="1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749" Sep 11 15:11:01.234456 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:01.234408 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749"} err="failed to get container status \"1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749\": rpc error: code = NotFound desc = could not find container \"1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749\": container with ID starting with 1d7276d1d4d33d665b9efdf7f4fba2a6de6133ba1020d3b363267ee905f69749 not found: ID does not exist" Sep 11 15:11:01.241473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:01.241446 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd9db7c5b-brpz9"] Sep 11 15:11:01.245759 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:01.245737 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fd9db7c5b-brpz9"] Sep 11 15:11:02.491524 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:02.491489 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1554405-2e32-47e6-95d5-2ccfabeaf476" path="/var/lib/kubelet/pods/a1554405-2e32-47e6-95d5-2ccfabeaf476/volumes" Sep 11 15:11:19.454708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:19.454678 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:19.473688 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:19.473662 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:20.318591 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:20.318561 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:37.560516 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.560428 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 15:11:37.561040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.560751 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="prometheus" containerID="cri-o://dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42" gracePeriod=600 Sep 11 15:11:37.561040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.560814 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy" containerID="cri-o://a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7" gracePeriod=600 Sep 11 15:11:37.561040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.560844 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="config-reloader" containerID="cri-o://19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273" gracePeriod=600 Sep 11 15:11:37.561040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.560892 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2" gracePeriod=600 Sep 11 15:11:37.561040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.560825 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="thanos-sidecar" containerID="cri-o://a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8" gracePeriod=600 Sep 11 15:11:37.561040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.560862 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy-web" containerID="cri-o://b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387" gracePeriod=600 Sep 11 15:11:37.804768 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.804744 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:37.936821 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.936768 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-thanos-prometheus-http-client-file\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.936999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.936829 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-config-out\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.936999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.936853 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-2\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.936999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.936872 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-config\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.936999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.936895 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-serving-certs-ca-bundle\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.936999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.936912 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.936999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.936936 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-kubelet-serving-ca-bundle\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.936999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.936964 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.936999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.936989 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-1\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937016 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-0\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937038 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-metrics-client-ca\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937073 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-grpc-tls\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937112 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-metrics-client-certs\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937151 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-kube-rbac-proxy\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937196 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-db\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937220 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-trusted-ca-bundle\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937245 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-tls\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937276 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-tls-assets\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937322 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr48d\" (UniqueName: \"kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-kube-api-access-fr48d\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937357 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:37.937403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937392 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-web-config\") pod \"974c5423-97e4-4180-8beb-741511108f83\" (UID: \"974c5423-97e4-4180-8beb-741511108f83\") " Sep 11 15:11:37.938243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937462 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:37.938243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937701 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:37.938243 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.937723 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:37.939179 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.939150 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:37.939486 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.939457 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:37.939581 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.939542 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:37.939678 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.939651 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:37.939738 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.939674 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-config" (OuterVolumeSpecName: "config") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:37.940020 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.939989 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:37.940239 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.940170 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:37.940499 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.940472 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-config-out" (OuterVolumeSpecName: "config-out") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:11:37.941311 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.940882 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:37.941311 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.940898 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:11:37.941311 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.941304 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:11:37.942128 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.942081 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:37.942253 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.942228 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:37.942659 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.942634 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:37.942751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.942731 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:11:37.942846 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.942820 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:37.943140 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.943126 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-kube-api-access-fr48d" (OuterVolumeSpecName: "kube-api-access-fr48d") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "kube-api-access-fr48d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:11:37.951996 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:37.951975 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-web-config" (OuterVolumeSpecName: "web-config") pod "974c5423-97e4-4180-8beb-741511108f83" (UID: "974c5423-97e4-4180-8beb-741511108f83"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:11:38.039037 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039014 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-web-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039037 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039036 2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039047 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-config-out\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039056 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039065 2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039073 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039082 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039091 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039100 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039108 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-configmap-metrics-client-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039116 2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-grpc-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039125 2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-metrics-client-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039132 2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-kube-rbac-proxy\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039140 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/974c5423-97e4-4180-8beb-741511108f83-prometheus-k8s-db\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039149 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/974c5423-97e4-4180-8beb-741511108f83-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039157 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/974c5423-97e4-4180-8beb-741511108f83-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039573 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039165 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-tls-assets\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.039573 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.039173 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr48d\" (UniqueName: \"kubernetes.io/projected/974c5423-97e4-4180-8beb-741511108f83-kube-api-access-fr48d\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:11:38.378374 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378346 2573 generic.go:356] "Generic (PLEG): container finished" podID="974c5423-97e4-4180-8beb-741511108f83" containerID="bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2" exitCode=0 Sep 11 15:11:38.378473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378376 2573 generic.go:356] "Generic (PLEG): container finished" podID="974c5423-97e4-4180-8beb-741511108f83" containerID="a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7" exitCode=0 Sep 11 15:11:38.378473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378398 2573 generic.go:356] "Generic (PLEG): container finished" podID="974c5423-97e4-4180-8beb-741511108f83" containerID="b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387" exitCode=0 Sep 11 15:11:38.378473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378408 2573 generic.go:356] "Generic (PLEG): container finished" podID="974c5423-97e4-4180-8beb-741511108f83" containerID="a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8" exitCode=0 Sep 11 15:11:38.378473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378421 2573 generic.go:356] "Generic (PLEG): container finished" podID="974c5423-97e4-4180-8beb-741511108f83" containerID="19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273" exitCode=0 Sep 11 15:11:38.378473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378429 2573 generic.go:356] "Generic (PLEG): container finished" podID="974c5423-97e4-4180-8beb-741511108f83" containerID="dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42" exitCode=0 Sep 11 15:11:38.378473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378430 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerDied","Data":"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2"} Sep 11 15:11:38.378473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378418 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.378848 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378512 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerDied","Data":"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7"} Sep 11 15:11:38.378848 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378571 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerDied","Data":"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387"} Sep 11 15:11:38.378848 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378594 2573 scope.go:117] "RemoveContainer" containerID="bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2" Sep 11 15:11:38.378848 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378617 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerDied","Data":"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8"} Sep 11 15:11:38.378848 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378659 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerDied","Data":"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273"} Sep 11 15:11:38.378848 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378701 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerDied","Data":"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42"} Sep 11 15:11:38.378848 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.378743 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"974c5423-97e4-4180-8beb-741511108f83","Type":"ContainerDied","Data":"9007456928cbea63ccf19fb223df09f3996acc144fd9463afec1b21e5dfc7a11"} Sep 11 15:11:38.380141 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:11:38.380113 2573 kuberuntime_gc.go:392] "Failed to remove container log dead symlink" err="remove /var/log/containers/prometheus-k8s-0_openshift-monitoring_kube-rbac-proxy-thanos-bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2.log: no such file or directory" path="/var/log/containers/prometheus-k8s-0_openshift-monitoring_kube-rbac-proxy-thanos-bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2.log" Sep 11 15:11:38.386530 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.386510 2573 scope.go:117] "RemoveContainer" containerID="a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7" Sep 11 15:11:38.393313 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.393296 2573 scope.go:117] "RemoveContainer" containerID="b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387" Sep 11 15:11:38.399651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.399641 2573 scope.go:117] "RemoveContainer" containerID="a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8" Sep 11 15:11:38.406126 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.406108 2573 scope.go:117] "RemoveContainer" containerID="19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273" Sep 11 15:11:38.408100 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.408080 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 15:11:38.412223 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.412065 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 15:11:38.415900 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.415876 2573 scope.go:117] "RemoveContainer" containerID="dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42" Sep 11 15:11:38.422607 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.422591 2573 scope.go:117] "RemoveContainer" containerID="9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37" Sep 11 15:11:38.428844 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.428830 2573 scope.go:117] "RemoveContainer" containerID="bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2" Sep 11 15:11:38.429115 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:11:38.429098 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": container with ID starting with bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2 not found: ID does not exist" containerID="bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2" Sep 11 15:11:38.429170 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.429121 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2"} err="failed to get container status \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": rpc error: code = NotFound desc = could not find container \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": container with ID starting with bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2 not found: ID does not exist" Sep 11 15:11:38.429170 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.429136 2573 scope.go:117] "RemoveContainer" containerID="a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7" Sep 11 15:11:38.429414 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:11:38.429395 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": container with ID starting with a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7 not found: ID does not exist" containerID="a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7" Sep 11 15:11:38.429453 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.429429 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7"} err="failed to get container status \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": rpc error: code = NotFound desc = could not find container \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": container with ID starting with a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7 not found: ID does not exist" Sep 11 15:11:38.429453 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.429446 2573 scope.go:117] "RemoveContainer" containerID="b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387" Sep 11 15:11:38.429687 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:11:38.429669 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": container with ID starting with b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387 not found: ID does not exist" containerID="b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387" Sep 11 15:11:38.429741 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.429691 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387"} err="failed to get container status \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": rpc error: code = NotFound desc = could not find container \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": container with ID starting with b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387 not found: ID does not exist" Sep 11 15:11:38.429741 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.429705 2573 scope.go:117] "RemoveContainer" containerID="a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8" Sep 11 15:11:38.430203 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:11:38.430188 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": container with ID starting with a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8 not found: ID does not exist" containerID="a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8" Sep 11 15:11:38.430248 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.430206 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8"} err="failed to get container status \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": rpc error: code = NotFound desc = could not find container \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": container with ID starting with a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8 not found: ID does not exist" Sep 11 15:11:38.430248 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.430220 2573 scope.go:117] "RemoveContainer" containerID="19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273" Sep 11 15:11:38.430463 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:11:38.430446 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": container with ID starting with 19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273 not found: ID does not exist" containerID="19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273" Sep 11 15:11:38.430503 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.430468 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273"} err="failed to get container status \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": rpc error: code = NotFound desc = could not find container \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": container with ID starting with 19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273 not found: ID does not exist" Sep 11 15:11:38.430503 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.430482 2573 scope.go:117] "RemoveContainer" containerID="dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42" Sep 11 15:11:38.430713 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:11:38.430697 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": container with ID starting with dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42 not found: ID does not exist" containerID="dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42" Sep 11 15:11:38.430751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.430716 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42"} err="failed to get container status \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": rpc error: code = NotFound desc = could not find container \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": container with ID starting with dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42 not found: ID does not exist" Sep 11 15:11:38.430751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.430735 2573 scope.go:117] "RemoveContainer" containerID="9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37" Sep 11 15:11:38.430984 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:11:38.430968 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": container with ID starting with 9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37 not found: ID does not exist" containerID="9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37" Sep 11 15:11:38.431030 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.430986 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37"} err="failed to get container status \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": rpc error: code = NotFound desc = could not find container \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": container with ID starting with 9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37 not found: ID does not exist" Sep 11 15:11:38.431030 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.430999 2573 scope.go:117] "RemoveContainer" containerID="bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2" Sep 11 15:11:38.431218 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.431203 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2"} err="failed to get container status \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": rpc error: code = NotFound desc = could not find container \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": container with ID starting with bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2 not found: ID does not exist" Sep 11 15:11:38.431270 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.431218 2573 scope.go:117] "RemoveContainer" containerID="a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7" Sep 11 15:11:38.431475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.431457 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7"} err="failed to get container status \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": rpc error: code = NotFound desc = could not find container \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": container with ID starting with a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7 not found: ID does not exist" Sep 11 15:11:38.431475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.431474 2573 scope.go:117] "RemoveContainer" containerID="b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387" Sep 11 15:11:38.431727 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.431707 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387"} err="failed to get container status \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": rpc error: code = NotFound desc = could not find container \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": container with ID starting with b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387 not found: ID does not exist" Sep 11 15:11:38.431823 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.431730 2573 scope.go:117] "RemoveContainer" containerID="a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8" Sep 11 15:11:38.432058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.432023 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8"} err="failed to get container status \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": rpc error: code = NotFound desc = could not find container \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": container with ID starting with a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8 not found: ID does not exist" Sep 11 15:11:38.432058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.432050 2573 scope.go:117] "RemoveContainer" containerID="19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273" Sep 11 15:11:38.432434 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.432324 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273"} err="failed to get container status \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": rpc error: code = NotFound desc = could not find container \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": container with ID starting with 19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273 not found: ID does not exist" Sep 11 15:11:38.432434 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.432349 2573 scope.go:117] "RemoveContainer" containerID="dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42" Sep 11 15:11:38.432641 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.432619 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42"} err="failed to get container status \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": rpc error: code = NotFound desc = could not find container \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": container with ID starting with dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42 not found: ID does not exist" Sep 11 15:11:38.432683 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.432642 2573 scope.go:117] "RemoveContainer" containerID="9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37" Sep 11 15:11:38.432921 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.432905 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37"} err="failed to get container status \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": rpc error: code = NotFound desc = could not find container \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": container with ID starting with 9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37 not found: ID does not exist" Sep 11 15:11:38.432994 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.432921 2573 scope.go:117] "RemoveContainer" containerID="bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2" Sep 11 15:11:38.433182 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.433163 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2"} err="failed to get container status \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": rpc error: code = NotFound desc = could not find container \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": container with ID starting with bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2 not found: ID does not exist" Sep 11 15:11:38.433226 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.433183 2573 scope.go:117] "RemoveContainer" containerID="a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7" Sep 11 15:11:38.433423 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.433404 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7"} err="failed to get container status \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": rpc error: code = NotFound desc = could not find container \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": container with ID starting with a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7 not found: ID does not exist" Sep 11 15:11:38.433467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.433424 2573 scope.go:117] "RemoveContainer" containerID="b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387" Sep 11 15:11:38.433639 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.433623 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387"} err="failed to get container status \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": rpc error: code = NotFound desc = could not find container \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": container with ID starting with b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387 not found: ID does not exist" Sep 11 15:11:38.433712 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.433640 2573 scope.go:117] "RemoveContainer" containerID="a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8" Sep 11 15:11:38.433884 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.433869 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8"} err="failed to get container status \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": rpc error: code = NotFound desc = could not find container \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": container with ID starting with a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8 not found: ID does not exist" Sep 11 15:11:38.433956 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.433884 2573 scope.go:117] "RemoveContainer" containerID="19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273" Sep 11 15:11:38.434135 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.434116 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273"} err="failed to get container status \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": rpc error: code = NotFound desc = could not find container \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": container with ID starting with 19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273 not found: ID does not exist" Sep 11 15:11:38.434184 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.434135 2573 scope.go:117] "RemoveContainer" containerID="dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42" Sep 11 15:11:38.434346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.434330 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42"} err="failed to get container status \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": rpc error: code = NotFound desc = could not find container \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": container with ID starting with dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42 not found: ID does not exist" Sep 11 15:11:38.434346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.434345 2573 scope.go:117] "RemoveContainer" containerID="9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37" Sep 11 15:11:38.434537 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.434523 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37"} err="failed to get container status \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": rpc error: code = NotFound desc = could not find container \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": container with ID starting with 9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37 not found: ID does not exist" Sep 11 15:11:38.434577 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.434537 2573 scope.go:117] "RemoveContainer" containerID="bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2" Sep 11 15:11:38.434758 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.434740 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2"} err="failed to get container status \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": rpc error: code = NotFound desc = could not find container \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": container with ID starting with bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2 not found: ID does not exist" Sep 11 15:11:38.434836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.434768 2573 scope.go:117] "RemoveContainer" containerID="a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7" Sep 11 15:11:38.435037 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.435016 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7"} err="failed to get container status \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": rpc error: code = NotFound desc = could not find container \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": container with ID starting with a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7 not found: ID does not exist" Sep 11 15:11:38.435077 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.435038 2573 scope.go:117] "RemoveContainer" containerID="b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387" Sep 11 15:11:38.435267 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.435251 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387"} err="failed to get container status \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": rpc error: code = NotFound desc = could not find container \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": container with ID starting with b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387 not found: ID does not exist" Sep 11 15:11:38.435318 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.435266 2573 scope.go:117] "RemoveContainer" containerID="a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8" Sep 11 15:11:38.435498 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.435469 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8"} err="failed to get container status \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": rpc error: code = NotFound desc = could not find container \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": container with ID starting with a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8 not found: ID does not exist" Sep 11 15:11:38.435498 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.435491 2573 scope.go:117] "RemoveContainer" containerID="19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273" Sep 11 15:11:38.435726 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.435708 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273"} err="failed to get container status \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": rpc error: code = NotFound desc = could not find container \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": container with ID starting with 19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273 not found: ID does not exist" Sep 11 15:11:38.435790 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.435728 2573 scope.go:117] "RemoveContainer" containerID="dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42" Sep 11 15:11:38.435990 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.435971 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42"} err="failed to get container status \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": rpc error: code = NotFound desc = could not find container \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": container with ID starting with dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42 not found: ID does not exist" Sep 11 15:11:38.436052 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.435991 2573 scope.go:117] "RemoveContainer" containerID="9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37" Sep 11 15:11:38.436224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.436208 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37"} err="failed to get container status \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": rpc error: code = NotFound desc = could not find container \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": container with ID starting with 9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37 not found: ID does not exist" Sep 11 15:11:38.436271 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.436226 2573 scope.go:117] "RemoveContainer" containerID="bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2" Sep 11 15:11:38.436419 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.436400 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2"} err="failed to get container status \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": rpc error: code = NotFound desc = could not find container \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": container with ID starting with bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2 not found: ID does not exist" Sep 11 15:11:38.436419 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.436417 2573 scope.go:117] "RemoveContainer" containerID="a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7" Sep 11 15:11:38.436653 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.436634 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7"} err="failed to get container status \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": rpc error: code = NotFound desc = could not find container \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": container with ID starting with a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7 not found: ID does not exist" Sep 11 15:11:38.436703 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.436655 2573 scope.go:117] "RemoveContainer" containerID="b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387" Sep 11 15:11:38.436917 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.436900 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387"} err="failed to get container status \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": rpc error: code = NotFound desc = could not find container \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": container with ID starting with b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387 not found: ID does not exist" Sep 11 15:11:38.436917 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.436916 2573 scope.go:117] "RemoveContainer" containerID="a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8" Sep 11 15:11:38.437141 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.437126 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8"} err="failed to get container status \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": rpc error: code = NotFound desc = could not find container \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": container with ID starting with a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8 not found: ID does not exist" Sep 11 15:11:38.437182 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.437141 2573 scope.go:117] "RemoveContainer" containerID="19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273" Sep 11 15:11:38.437360 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.437341 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273"} err="failed to get container status \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": rpc error: code = NotFound desc = could not find container \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": container with ID starting with 19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273 not found: ID does not exist" Sep 11 15:11:38.437407 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.437361 2573 scope.go:117] "RemoveContainer" containerID="dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42" Sep 11 15:11:38.437581 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.437562 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42"} err="failed to get container status \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": rpc error: code = NotFound desc = could not find container \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": container with ID starting with dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42 not found: ID does not exist" Sep 11 15:11:38.437638 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.437582 2573 scope.go:117] "RemoveContainer" containerID="9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37" Sep 11 15:11:38.437829 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.437812 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37"} err="failed to get container status \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": rpc error: code = NotFound desc = could not find container \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": container with ID starting with 9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37 not found: ID does not exist" Sep 11 15:11:38.437829 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.437829 2573 scope.go:117] "RemoveContainer" containerID="bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2" Sep 11 15:11:38.438052 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.438037 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2"} err="failed to get container status \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": rpc error: code = NotFound desc = could not find container \"bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2\": container with ID starting with bd6b4f8f832c883eb544623fdcdb0698ba2b8224dc66b7a8a521b67ceaffacc2 not found: ID does not exist" Sep 11 15:11:38.438052 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.438052 2573 scope.go:117] "RemoveContainer" containerID="a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7" Sep 11 15:11:38.438291 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.438274 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7"} err="failed to get container status \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": rpc error: code = NotFound desc = could not find container \"a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7\": container with ID starting with a2864b6c43741d1b96f015bd4b4304495f58cb3aa0476a3cfef8961cb29851b7 not found: ID does not exist" Sep 11 15:11:38.438337 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.438291 2573 scope.go:117] "RemoveContainer" containerID="b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387" Sep 11 15:11:38.438520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.438504 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387"} err="failed to get container status \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": rpc error: code = NotFound desc = could not find container \"b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387\": container with ID starting with b6d0020e4b186a89fcf6054b60dc7aa4eabff04fd08cae2110f1274684972387 not found: ID does not exist" Sep 11 15:11:38.438558 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.438519 2573 scope.go:117] "RemoveContainer" containerID="a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8" Sep 11 15:11:38.438723 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.438707 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8"} err="failed to get container status \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": rpc error: code = NotFound desc = could not find container \"a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8\": container with ID starting with a79cc2bf794a075c8c2b2798224fc027d156345a513ede178d140ace028d04a8 not found: ID does not exist" Sep 11 15:11:38.438799 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.438723 2573 scope.go:117] "RemoveContainer" containerID="19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273" Sep 11 15:11:38.438940 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.438924 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273"} err="failed to get container status \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": rpc error: code = NotFound desc = could not find container \"19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273\": container with ID starting with 19295cd8c3007a1deddbe43300275f79646d5afa0e77cfcffd1fc486b48f9273 not found: ID does not exist" Sep 11 15:11:38.438981 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.438940 2573 scope.go:117] "RemoveContainer" containerID="dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42" Sep 11 15:11:38.439153 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.439138 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42"} err="failed to get container status \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": rpc error: code = NotFound desc = could not find container \"dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42\": container with ID starting with dd01239c454c496bc53b1515fc0231249f910430e5d7ce3084f4a26aa938bd42 not found: ID does not exist" Sep 11 15:11:38.439217 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.439154 2573 scope.go:117] "RemoveContainer" containerID="9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37" Sep 11 15:11:38.439397 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.439380 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37"} err="failed to get container status \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": rpc error: code = NotFound desc = could not find container \"9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37\": container with ID starting with 9d047d14e26d96e21bcb769ebafd3b4f09152ef5813f7d77265ec3160bfbec37 not found: ID does not exist" Sep 11 15:11:38.443257 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443238 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 15:11:38.443530 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443518 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy-web" Sep 11 15:11:38.443572 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443533 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy-web" Sep 11 15:11:38.443572 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443544 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="init-config-reloader" Sep 11 15:11:38.443572 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443549 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="init-config-reloader" Sep 11 15:11:38.443669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443558 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy" Sep 11 15:11:38.443669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443594 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy" Sep 11 15:11:38.443669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443603 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1554405-2e32-47e6-95d5-2ccfabeaf476" containerName="console" Sep 11 15:11:38.443669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443608 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1554405-2e32-47e6-95d5-2ccfabeaf476" containerName="console" Sep 11 15:11:38.443669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443617 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="config-reloader" Sep 11 15:11:38.443669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443635 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="config-reloader" Sep 11 15:11:38.443669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443643 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="prometheus" Sep 11 15:11:38.443669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443648 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="prometheus" Sep 11 15:11:38.443669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443664 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="thanos-sidecar" Sep 11 15:11:38.443669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443670 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="thanos-sidecar" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443676 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy-thanos" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443682 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy-thanos" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443689 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6edb816-b6c2-4eb8-906f-250092271b4f" containerName="console" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443694 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6edb816-b6c2-4eb8-906f-250092271b4f" containerName="console" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443748 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6edb816-b6c2-4eb8-906f-250092271b4f" containerName="console" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443758 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="prometheus" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443766 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1554405-2e32-47e6-95d5-2ccfabeaf476" containerName="console" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443796 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="config-reloader" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443802 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy-web" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443808 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443814 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="thanos-sidecar" Sep 11 15:11:38.444045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.443822 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="974c5423-97e4-4180-8beb-741511108f83" containerName="kube-rbac-proxy-thanos" Sep 11 15:11:38.449000 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.448983 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.451539 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.451517 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 15:11:38.451637 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.451560 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 15:11:38.451637 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.451573 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 15:11:38.451885 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.451768 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9prk4\"" Sep 11 15:11:38.451885 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.451831 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 15:11:38.451885 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.451847 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 15:11:38.452080 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.451897 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 15:11:38.452080 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.451930 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 15:11:38.452080 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.451941 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 15:11:38.452573 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.452554 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 15:11:38.452673 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.452578 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 15:11:38.452673 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.452557 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dd9sqqrbimsvq\"" Sep 11 15:11:38.452673 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.452648 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 15:11:38.452925 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.452912 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 15:11:38.453667 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.453650 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 15:11:38.456245 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.456225 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 15:11:38.457688 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.457669 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 15:11:38.466004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.465986 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 15:11:38.497985 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.497955 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="974c5423-97e4-4180-8beb-741511108f83" path="/var/lib/kubelet/pods/974c5423-97e4-4180-8beb-741511108f83/volumes" Sep 11 15:11:38.543465 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543446 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-config\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543658 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-web-config\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbc6g\" (UniqueName: \"kubernetes.io/projected/d355c750-5d72-4d8c-9b43-62eb49b733bb-kube-api-access-xbc6g\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543810 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.543827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543829 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.544118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543853 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.544118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d355c750-5d72-4d8c-9b43-62eb49b733bb-config-out\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.544118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.544118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543920 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.544118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.544118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.544118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.543973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d355c750-5d72-4d8c-9b43-62eb49b733bb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.644303 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.644303 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.644691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.644691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644348 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d355c750-5d72-4d8c-9b43-62eb49b733bb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.644691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-config\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.644691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.644691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.644691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644571 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.644691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.644691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644694 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-web-config\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644845 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbc6g\" (UniqueName: \"kubernetes.io/projected/d355c750-5d72-4d8c-9b43-62eb49b733bb-kube-api-access-xbc6g\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644948 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644957 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.644983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d355c750-5d72-4d8c-9b43-62eb49b733bb-config-out\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.645007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645636 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.645461 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.645636 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.645532 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.647229 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.646347 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.647953 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.647609 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d355c750-5d72-4d8c-9b43-62eb49b733bb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.647953 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.647858 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.648248 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.648216 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.648491 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.648462 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.648568 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.648519 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.649221 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.649048 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.649221 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.649107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.649429 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.649216 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-config\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.649555 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.649507 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.649657 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.649611 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d355c750-5d72-4d8c-9b43-62eb49b733bb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.650399 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.650375 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.650494 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.650480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.651309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.651282 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-web-config\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.651383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.651345 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d355c750-5d72-4d8c-9b43-62eb49b733bb-config-out\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.651655 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.651637 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d355c750-5d72-4d8c-9b43-62eb49b733bb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.654986 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.654970 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbc6g\" (UniqueName: \"kubernetes.io/projected/d355c750-5d72-4d8c-9b43-62eb49b733bb-kube-api-access-xbc6g\") pod \"prometheus-k8s-0\" (UID: \"d355c750-5d72-4d8c-9b43-62eb49b733bb\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:38.758109 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:38.758081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:39.044441 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:39.044406 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 15:11:39.046199 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:11:39.046175 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd355c750_5d72_4d8c_9b43_62eb49b733bb.slice/crio-5b58bbe3b93adc2b3e140b57af6762c0be47364279aa284797dccb3efd57075e WatchSource:0}: Error finding container 5b58bbe3b93adc2b3e140b57af6762c0be47364279aa284797dccb3efd57075e: Status 404 returned error can't find the container with id 5b58bbe3b93adc2b3e140b57af6762c0be47364279aa284797dccb3efd57075e Sep 11 15:11:39.384300 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:39.384269 2573 generic.go:356] "Generic (PLEG): container finished" podID="d355c750-5d72-4d8c-9b43-62eb49b733bb" containerID="c14d6f05b712900a5b0913f8a205caa636025ec6ebab24dfcd412cdd96391d2d" exitCode=0 Sep 11 15:11:39.384426 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:39.384359 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d355c750-5d72-4d8c-9b43-62eb49b733bb","Type":"ContainerDied","Data":"c14d6f05b712900a5b0913f8a205caa636025ec6ebab24dfcd412cdd96391d2d"} Sep 11 15:11:39.384493 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:39.384437 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d355c750-5d72-4d8c-9b43-62eb49b733bb","Type":"ContainerStarted","Data":"5b58bbe3b93adc2b3e140b57af6762c0be47364279aa284797dccb3efd57075e"} Sep 11 15:11:40.393681 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:40.393639 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d355c750-5d72-4d8c-9b43-62eb49b733bb","Type":"ContainerStarted","Data":"b7855c9758e69970fdec0c4945f8744f60a9a039197a3e9bcfc9614ed3909c7f"} Sep 11 15:11:40.394070 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:40.393722 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d355c750-5d72-4d8c-9b43-62eb49b733bb","Type":"ContainerStarted","Data":"0b8c123a41852c2a0abd7350f0cdc647f7338d639543f6b7e0349f2250049852"} Sep 11 15:11:40.394070 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:40.393753 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d355c750-5d72-4d8c-9b43-62eb49b733bb","Type":"ContainerStarted","Data":"bb5ffbc37e1912d139a385ad5f5bed30ef8ae3fb1301e71aa79d013758a4142e"} Sep 11 15:11:40.394070 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:40.393799 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d355c750-5d72-4d8c-9b43-62eb49b733bb","Type":"ContainerStarted","Data":"6c3c96f7f070058d8ddc40aa5d746ca094945021496904d38d9735df2371fd3d"} Sep 11 15:11:40.394070 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:40.393827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d355c750-5d72-4d8c-9b43-62eb49b733bb","Type":"ContainerStarted","Data":"8d9a391ae025d0d2174dab8da15718afb5986f483968b10563ca642f347bfd40"} Sep 11 15:11:40.394070 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:40.393850 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d355c750-5d72-4d8c-9b43-62eb49b733bb","Type":"ContainerStarted","Data":"309aa46c63d3dfa00ea463ca6db7e87b47c939373600592f30386e997add9b9f"} Sep 11 15:11:40.430120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:40.430042 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.411747504 podStartE2EDuration="2.430020544s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.27304ms" isStatefulPod=false podCreationTimestamp="2026-09-11 15:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:11:40.428430812 +0000 UTC m=+182.473755792" watchObservedRunningTime="2026-09-11 15:11:40.430020544 +0000 UTC m=+182.475345523" Sep 11 15:11:43.758801 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:43.758753 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:11:51.156767 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:11:51.156732 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d7bdc59-2qqkv"] Sep 11 15:12:16.176345 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.176300 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d7bdc59-2qqkv" podUID="c5e68cba-91fa-45a1-a639-034e9f9af6ef" containerName="console" containerID="cri-o://b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519" gracePeriod=15 Sep 11 15:12:16.408985 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.408962 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:12:16.519349 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519287 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-oauth-serving-cert\") pod \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " Sep 11 15:12:16.519349 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519320 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-serving-cert\") pod \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " Sep 11 15:12:16.519537 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519357 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-config\") pod \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " Sep 11 15:12:16.519537 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519376 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-trusted-ca-bundle\") pod \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " Sep 11 15:12:16.519537 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519406 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-oauth-config\") pod \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " Sep 11 15:12:16.519537 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519422 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7mrh\" (UniqueName: \"kubernetes.io/projected/c5e68cba-91fa-45a1-a639-034e9f9af6ef-kube-api-access-w7mrh\") pod \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " Sep 11 15:12:16.519537 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519436 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-service-ca\") pod \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\" (UID: \"c5e68cba-91fa-45a1-a639-034e9f9af6ef\") " Sep 11 15:12:16.519819 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519766 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c5e68cba-91fa-45a1-a639-034e9f9af6ef" (UID: "c5e68cba-91fa-45a1-a639-034e9f9af6ef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:12:16.519888 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519851 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-config" (OuterVolumeSpecName: "console-config") pod "c5e68cba-91fa-45a1-a639-034e9f9af6ef" (UID: "c5e68cba-91fa-45a1-a639-034e9f9af6ef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:12:16.519955 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519925 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-service-ca" (OuterVolumeSpecName: "service-ca") pod "c5e68cba-91fa-45a1-a639-034e9f9af6ef" (UID: "c5e68cba-91fa-45a1-a639-034e9f9af6ef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:12:16.520007 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.519946 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c5e68cba-91fa-45a1-a639-034e9f9af6ef" (UID: "c5e68cba-91fa-45a1-a639-034e9f9af6ef"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:12:16.521763 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.521741 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e68cba-91fa-45a1-a639-034e9f9af6ef-kube-api-access-w7mrh" (OuterVolumeSpecName: "kube-api-access-w7mrh") pod "c5e68cba-91fa-45a1-a639-034e9f9af6ef" (UID: "c5e68cba-91fa-45a1-a639-034e9f9af6ef"). InnerVolumeSpecName "kube-api-access-w7mrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:12:16.522161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.522134 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c5e68cba-91fa-45a1-a639-034e9f9af6ef" (UID: "c5e68cba-91fa-45a1-a639-034e9f9af6ef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:12:16.522161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.522149 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c5e68cba-91fa-45a1-a639-034e9f9af6ef" (UID: "c5e68cba-91fa-45a1-a639-034e9f9af6ef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:12:16.530958 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.530933 2573 generic.go:356] "Generic (PLEG): container finished" podID="c5e68cba-91fa-45a1-a639-034e9f9af6ef" containerID="b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519" exitCode=0 Sep 11 15:12:16.531045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.530975 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7bdc59-2qqkv" Sep 11 15:12:16.531045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.530989 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7bdc59-2qqkv" event={"ID":"c5e68cba-91fa-45a1-a639-034e9f9af6ef","Type":"ContainerDied","Data":"b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519"} Sep 11 15:12:16.531045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.531025 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7bdc59-2qqkv" event={"ID":"c5e68cba-91fa-45a1-a639-034e9f9af6ef","Type":"ContainerDied","Data":"03f616e3678fbd8efb48332f14f12abf0411db16b0705c70f2c29208f6d4e352"} Sep 11 15:12:16.531045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.531044 2573 scope.go:117] "RemoveContainer" containerID="b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519" Sep 11 15:12:16.539814 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.539800 2573 scope.go:117] "RemoveContainer" containerID="b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519" Sep 11 15:12:16.540083 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:12:16.540065 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519\": container with ID starting with b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519 not found: ID does not exist" containerID="b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519" Sep 11 15:12:16.540146 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.540104 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519"} err="failed to get container status \"b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519\": rpc error: code = NotFound desc = could not find container \"b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519\": container with ID starting with b82556ebb9a5df42a93132bfc934a41627d977f55158e28179d0b23fcb588519 not found: ID does not exist" Sep 11 15:12:16.552691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.552671 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d7bdc59-2qqkv"] Sep 11 15:12:16.556452 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.556432 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d7bdc59-2qqkv"] Sep 11 15:12:16.620945 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.620927 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:12:16.620945 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.620946 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-oauth-serving-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:12:16.621054 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.620954 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-serving-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:12:16.621054 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.620963 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:12:16.621054 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.620972 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5e68cba-91fa-45a1-a639-034e9f9af6ef-trusted-ca-bundle\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:12:16.621054 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.620980 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5e68cba-91fa-45a1-a639-034e9f9af6ef-console-oauth-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:12:16.621054 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:16.620989 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7mrh\" (UniqueName: \"kubernetes.io/projected/c5e68cba-91fa-45a1-a639-034e9f9af6ef-kube-api-access-w7mrh\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:12:18.492031 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:18.491991 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5e68cba-91fa-45a1-a639-034e9f9af6ef" path="/var/lib/kubelet/pods/c5e68cba-91fa-45a1-a639-034e9f9af6ef/volumes" Sep 11 15:12:38.759041 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:38.758972 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:12:38.773665 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:38.773639 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:12:39.633093 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:12:39.633070 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 15:13:38.368705 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:38.368674 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:13:38.374367 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:38.374344 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:13:38.380401 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:38.380380 2573 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 15:13:56.516228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.516199 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s"] Sep 11 15:13:56.518666 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.516489 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e68cba-91fa-45a1-a639-034e9f9af6ef" containerName="console" Sep 11 15:13:56.518666 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.516500 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e68cba-91fa-45a1-a639-034e9f9af6ef" containerName="console" Sep 11 15:13:56.518666 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.516557 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e68cba-91fa-45a1-a639-034e9f9af6ef" containerName="console" Sep 11 15:13:56.519517 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.519503 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.523261 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.523239 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 15:13:56.523379 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.523249 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 15:13:56.523428 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.523384 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 15:13:56.524257 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.524239 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 15:13:56.524359 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.524278 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:13:56.525547 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.525525 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xqpmn\"" Sep 11 15:13:56.535871 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.535850 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s"] Sep 11 15:13:56.629165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.629137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smz9d\" (UniqueName: \"kubernetes.io/projected/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-kube-api-access-smz9d\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.629266 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.629170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-manager-config\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.629266 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.629191 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-cert\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.629266 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.629261 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-metrics-cert\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.730494 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.730470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smz9d\" (UniqueName: \"kubernetes.io/projected/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-kube-api-access-smz9d\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.730576 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.730502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-manager-config\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.730576 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.730523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-cert\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.730576 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.730560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-metrics-cert\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.731184 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.731159 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-manager-config\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.733257 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.733226 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-metrics-cert\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.733347 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.733286 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-cert\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.739752 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.739735 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smz9d\" (UniqueName: \"kubernetes.io/projected/2ab062ec-1aba-4b5e-9c88-806ba856d0bb-kube-api-access-smz9d\") pod \"lws-controller-manager-68f4698bc6-gxs2s\" (UID: \"2ab062ec-1aba-4b5e-9c88-806ba856d0bb\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:56.828761 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:56.828733 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:57.121144 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:57.121042 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s"] Sep 11 15:13:57.123678 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:13:57.123651 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab062ec_1aba_4b5e_9c88_806ba856d0bb.slice/crio-b5a1d9410f929897c9675679af5958ab5d1f9d9b89b096540f3d840660ad04c6 WatchSource:0}: Error finding container b5a1d9410f929897c9675679af5958ab5d1f9d9b89b096540f3d840660ad04c6: Status 404 returned error can't find the container with id b5a1d9410f929897c9675679af5958ab5d1f9d9b89b096540f3d840660ad04c6 Sep 11 15:13:57.125678 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:57.125660 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:13:57.911962 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:57.911823 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" event={"ID":"2ab062ec-1aba-4b5e-9c88-806ba856d0bb","Type":"ContainerStarted","Data":"b5a1d9410f929897c9675679af5958ab5d1f9d9b89b096540f3d840660ad04c6"} Sep 11 15:13:59.923666 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:59.923637 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" event={"ID":"2ab062ec-1aba-4b5e-9c88-806ba856d0bb","Type":"ContainerStarted","Data":"27244b44ec52d01f051e14deccb5884a37350e0fda0e6e499fd2718ab9fe1dcd"} Sep 11 15:13:59.924030 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:59.923767 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:13:59.941094 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:13:59.941049 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" podStartSLOduration=1.500163271 podStartE2EDuration="3.941036554s" totalImagesPullingTime="2.440873283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:13:59.93964858 +0000 UTC m=+321.984973585" watchObservedRunningTime="2026-09-11 15:13:59.941036554 +0000 UTC m=+321.986361533" Sep 11 15:14:05.304345 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.304308 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm"] Sep 11 15:14:05.307648 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.307631 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.310388 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.310366 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 15:14:05.310706 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.310690 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 15:14:05.310765 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.310689 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gkvb7\"" Sep 11 15:14:05.310765 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.310755 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 15:14:05.311061 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.311048 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 15:14:05.324703 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.324677 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm"] Sep 11 15:14:05.402815 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.402792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a5a4890-3732-462d-b1cb-d0b127d59b97-webhook-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.402920 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.402839 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6a5a4890-3732-462d-b1cb-d0b127d59b97-manifests\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.402920 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.402876 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5a4890-3732-462d-b1cb-d0b127d59b97-tmp\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.402920 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.402908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdw6l\" (UniqueName: \"kubernetes.io/projected/6a5a4890-3732-462d-b1cb-d0b127d59b97-kube-api-access-fdw6l\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.403050 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.402976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6a5a4890-3732-462d-b1cb-d0b127d59b97-apiservice-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.503719 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.503691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6a5a4890-3732-462d-b1cb-d0b127d59b97-manifests\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.503820 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.503727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5a4890-3732-462d-b1cb-d0b127d59b97-tmp\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.503820 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.503754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdw6l\" (UniqueName: \"kubernetes.io/projected/6a5a4890-3732-462d-b1cb-d0b127d59b97-kube-api-access-fdw6l\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.503820 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.503813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6a5a4890-3732-462d-b1cb-d0b127d59b97-apiservice-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.503950 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.503880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a5a4890-3732-462d-b1cb-d0b127d59b97-webhook-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.504157 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.504134 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6a5a4890-3732-462d-b1cb-d0b127d59b97-manifests\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.506176 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.506158 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5a4890-3732-462d-b1cb-d0b127d59b97-tmp\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.506389 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.506371 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a5a4890-3732-462d-b1cb-d0b127d59b97-webhook-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.506538 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.506518 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6a5a4890-3732-462d-b1cb-d0b127d59b97-apiservice-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.512512 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.512491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdw6l\" (UniqueName: \"kubernetes.io/projected/6a5a4890-3732-462d-b1cb-d0b127d59b97-kube-api-access-fdw6l\") pod \"opendatahub-operator-controller-manager-87954dbdf-zmdnm\" (UID: \"6a5a4890-3732-462d-b1cb-d0b127d59b97\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.618154 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.618126 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:05.901145 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.901121 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm"] Sep 11 15:14:05.903443 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:14:05.903417 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5a4890_3732_462d_b1cb_d0b127d59b97.slice/crio-7cfcc1ccdb89dbadf176da1b1131c6de10acc3930659674287b453480a2e5426 WatchSource:0}: Error finding container 7cfcc1ccdb89dbadf176da1b1131c6de10acc3930659674287b453480a2e5426: Status 404 returned error can't find the container with id 7cfcc1ccdb89dbadf176da1b1131c6de10acc3930659674287b453480a2e5426 Sep 11 15:14:05.949587 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:05.949565 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" event={"ID":"6a5a4890-3732-462d-b1cb-d0b127d59b97","Type":"ContainerStarted","Data":"7cfcc1ccdb89dbadf176da1b1131c6de10acc3930659674287b453480a2e5426"} Sep 11 15:14:09.968791 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:09.968733 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" event={"ID":"6a5a4890-3732-462d-b1cb-d0b127d59b97","Type":"ContainerStarted","Data":"e93c497234cc012701ea07aba4ca33e670937d917093eaeaf0e4c219334f194e"} Sep 11 15:14:10.929634 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:10.929605 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-gxs2s" Sep 11 15:14:10.975641 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:10.975601 2573 generic.go:356] "Generic (PLEG): container finished" podID="6a5a4890-3732-462d-b1cb-d0b127d59b97" containerID="e93c497234cc012701ea07aba4ca33e670937d917093eaeaf0e4c219334f194e" exitCode=0 Sep 11 15:14:10.976115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:10.975689 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" event={"ID":"6a5a4890-3732-462d-b1cb-d0b127d59b97","Type":"ContainerDied","Data":"e93c497234cc012701ea07aba4ca33e670937d917093eaeaf0e4c219334f194e"} Sep 11 15:14:11.981934 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:11.981902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" event={"ID":"6a5a4890-3732-462d-b1cb-d0b127d59b97","Type":"ContainerStarted","Data":"dd13b4e93fea67f25378a448d9ebb481407e7ae28964ebb0df170a08ec73cc49"} Sep 11 15:14:11.982294 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:11.982047 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:12.007584 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:12.007539 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" podStartSLOduration=2.541182689 podStartE2EDuration="7.007526677s" totalImagesPullingTime="3.476872924s" totalInitContainerRuntime="989.471064ms" isStatefulPod=false podCreationTimestamp="2026-09-11 15:14:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:14:12.007313425 +0000 UTC m=+334.052638407" watchObservedRunningTime="2026-09-11 15:14:12.007526677 +0000 UTC m=+334.052851656" Sep 11 15:14:22.988232 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:22.988200 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-zmdnm" Sep 11 15:14:25.800868 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.800835 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5"] Sep 11 15:14:25.816068 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.816044 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:25.818903 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.818880 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 15:14:25.819180 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.819163 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 15:14:25.819317 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.819290 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 15:14:25.819435 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.819369 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 15:14:25.819859 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.819843 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9v2hd\"" Sep 11 15:14:25.829652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.829634 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5"] Sep 11 15:14:25.969048 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.969019 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03e58a3-53d9-435f-911a-a8c5922efe4d-tls-certs\") pod \"kube-auth-proxy-85b8bff76-l9bt5\" (UID: \"f03e58a3-53d9-435f-911a-a8c5922efe4d\") " pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:25.969171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.969053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x29nb\" (UniqueName: \"kubernetes.io/projected/f03e58a3-53d9-435f-911a-a8c5922efe4d-kube-api-access-x29nb\") pod \"kube-auth-proxy-85b8bff76-l9bt5\" (UID: \"f03e58a3-53d9-435f-911a-a8c5922efe4d\") " pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:25.969171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:25.969074 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f03e58a3-53d9-435f-911a-a8c5922efe4d-tmp\") pod \"kube-auth-proxy-85b8bff76-l9bt5\" (UID: \"f03e58a3-53d9-435f-911a-a8c5922efe4d\") " pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:26.070314 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:26.070256 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03e58a3-53d9-435f-911a-a8c5922efe4d-tls-certs\") pod \"kube-auth-proxy-85b8bff76-l9bt5\" (UID: \"f03e58a3-53d9-435f-911a-a8c5922efe4d\") " pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:26.070314 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:26.070285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x29nb\" (UniqueName: \"kubernetes.io/projected/f03e58a3-53d9-435f-911a-a8c5922efe4d-kube-api-access-x29nb\") pod \"kube-auth-proxy-85b8bff76-l9bt5\" (UID: \"f03e58a3-53d9-435f-911a-a8c5922efe4d\") " pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:26.070314 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:26.070304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f03e58a3-53d9-435f-911a-a8c5922efe4d-tmp\") pod \"kube-auth-proxy-85b8bff76-l9bt5\" (UID: \"f03e58a3-53d9-435f-911a-a8c5922efe4d\") " pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:26.072604 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:26.072584 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f03e58a3-53d9-435f-911a-a8c5922efe4d-tmp\") pod \"kube-auth-proxy-85b8bff76-l9bt5\" (UID: \"f03e58a3-53d9-435f-911a-a8c5922efe4d\") " pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:26.072847 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:26.072831 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03e58a3-53d9-435f-911a-a8c5922efe4d-tls-certs\") pod \"kube-auth-proxy-85b8bff76-l9bt5\" (UID: \"f03e58a3-53d9-435f-911a-a8c5922efe4d\") " pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:26.081539 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:26.081518 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x29nb\" (UniqueName: \"kubernetes.io/projected/f03e58a3-53d9-435f-911a-a8c5922efe4d-kube-api-access-x29nb\") pod \"kube-auth-proxy-85b8bff76-l9bt5\" (UID: \"f03e58a3-53d9-435f-911a-a8c5922efe4d\") " pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:26.125455 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:26.125435 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" Sep 11 15:14:26.398739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:26.398715 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5"] Sep 11 15:14:26.401127 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:14:26.401102 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03e58a3_53d9_435f_911a_a8c5922efe4d.slice/crio-d2f91f18f387d936fccb7854cbdf5bb259e8695808d8ae2e52203b59ab5ba99c WatchSource:0}: Error finding container d2f91f18f387d936fccb7854cbdf5bb259e8695808d8ae2e52203b59ab5ba99c: Status 404 returned error can't find the container with id d2f91f18f387d936fccb7854cbdf5bb259e8695808d8ae2e52203b59ab5ba99c Sep 11 15:14:27.047511 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:27.047441 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" event={"ID":"f03e58a3-53d9-435f-911a-a8c5922efe4d","Type":"ContainerStarted","Data":"d2f91f18f387d936fccb7854cbdf5bb259e8695808d8ae2e52203b59ab5ba99c"} Sep 11 15:14:30.065079 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:30.064999 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" event={"ID":"f03e58a3-53d9-435f-911a-a8c5922efe4d","Type":"ContainerStarted","Data":"2c37b19ac560497263decd00275e270d82fee72fecfc90e3b492d45215cc67b1"} Sep 11 15:14:30.082192 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:14:30.082148 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85b8bff76-l9bt5" podStartSLOduration=2.284178822 podStartE2EDuration="5.082132285s" totalImagesPullingTime="2.797953463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:14:30.080032997 +0000 UTC m=+352.125357977" watchObservedRunningTime="2026-09-11 15:14:30.082132285 +0000 UTC m=+352.127457265" Sep 11 15:15:05.745671 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.745607 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cd998cc94-db6l7"] Sep 11 15:15:05.752045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.752019 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.754602 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.754581 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 15:15:05.755546 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.755523 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wnkdc\"" Sep 11 15:15:05.755797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.755647 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 15:15:05.755797 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.755542 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 15:15:05.756081 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.756063 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 15:15:05.756195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.756075 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 15:15:05.756195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.756080 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 15:15:05.756409 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.756375 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 15:15:05.761242 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.761026 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd998cc94-db6l7"] Sep 11 15:15:05.765166 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.765141 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 15:15:05.873108 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.873080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-oauth-serving-cert\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.873242 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.873131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-console-config\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.873242 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.873156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-trusted-ca-bundle\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.873242 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.873218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-service-ca\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.873377 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.873281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-console-oauth-config\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.873377 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.873347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-console-serving-cert\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.873449 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.873384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k684s\" (UniqueName: \"kubernetes.io/projected/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-kube-api-access-k684s\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.974638 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.974618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k684s\" (UniqueName: \"kubernetes.io/projected/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-kube-api-access-k684s\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.974755 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.974658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-oauth-serving-cert\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.974755 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.974682 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-console-config\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.974755 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.974697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-trusted-ca-bundle\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.974961 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.974820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-service-ca\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.974961 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.974849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-console-oauth-config\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.974961 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.974885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-console-serving-cert\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.975437 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.975417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-oauth-serving-cert\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.975496 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.975445 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-console-config\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.975600 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.975573 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-service-ca\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.975650 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.975617 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-trusted-ca-bundle\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.977521 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.977498 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-console-oauth-config\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.977611 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.977594 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-console-serving-cert\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:05.982960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:05.982940 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k684s\" (UniqueName: \"kubernetes.io/projected/fc367530-f4f4-4a1d-9cdd-046e94e2f1d3-kube-api-access-k684s\") pod \"console-7cd998cc94-db6l7\" (UID: \"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3\") " pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:06.065432 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:06.065374 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:06.338010 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:06.337982 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd998cc94-db6l7"] Sep 11 15:15:06.340827 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:15:06.340797 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc367530_f4f4_4a1d_9cdd_046e94e2f1d3.slice/crio-70cc278e8d3b038a60da899a9a40f8f61e2b4eeed8f08e4ae97b2e5312a3e089 WatchSource:0}: Error finding container 70cc278e8d3b038a60da899a9a40f8f61e2b4eeed8f08e4ae97b2e5312a3e089: Status 404 returned error can't find the container with id 70cc278e8d3b038a60da899a9a40f8f61e2b4eeed8f08e4ae97b2e5312a3e089 Sep 11 15:15:07.221431 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:07.221397 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd998cc94-db6l7" event={"ID":"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3","Type":"ContainerStarted","Data":"993abbf0f6671d68fc85a3c9bad9bc8c10892f5264975108fc8c1522f88ce63d"} Sep 11 15:15:07.221820 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:07.221449 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd998cc94-db6l7" event={"ID":"fc367530-f4f4-4a1d-9cdd-046e94e2f1d3","Type":"ContainerStarted","Data":"70cc278e8d3b038a60da899a9a40f8f61e2b4eeed8f08e4ae97b2e5312a3e089"} Sep 11 15:15:07.240618 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:07.240570 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cd998cc94-db6l7" podStartSLOduration=2.240556236 podStartE2EDuration="2.240556236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:15:07.239638933 +0000 UTC m=+389.284963914" watchObservedRunningTime="2026-09-11 15:15:07.240556236 +0000 UTC m=+389.285881267" Sep 11 15:15:16.065625 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:16.065588 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:16.066195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:16.065641 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:16.070988 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:16.070968 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:15:16.266601 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:15:16.266569 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cd998cc94-db6l7" Sep 11 15:16:03.753982 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:03.753898 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8"] Sep 11 15:16:03.757207 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:03.757190 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:03.760296 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:03.760276 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 15:16:03.761154 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:03.761128 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-cp9bf\"" Sep 11 15:16:03.761261 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:03.761184 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 15:16:03.772157 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:03.772133 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8"] Sep 11 15:16:03.800294 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:03.800267 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5f5q\" (UniqueName: \"kubernetes.io/projected/68a706e9-72b8-4bb4-887a-0ea8044ae473-kube-api-access-v5f5q\") pod \"limitador-operator-controller-manager-74495c69f8-84lb8\" (UID: \"68a706e9-72b8-4bb4-887a-0ea8044ae473\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:03.901239 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:03.901209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5f5q\" (UniqueName: \"kubernetes.io/projected/68a706e9-72b8-4bb4-887a-0ea8044ae473-kube-api-access-v5f5q\") pod \"limitador-operator-controller-manager-74495c69f8-84lb8\" (UID: \"68a706e9-72b8-4bb4-887a-0ea8044ae473\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:03.921785 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:03.921735 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5f5q\" (UniqueName: \"kubernetes.io/projected/68a706e9-72b8-4bb4-887a-0ea8044ae473-kube-api-access-v5f5q\") pod \"limitador-operator-controller-manager-74495c69f8-84lb8\" (UID: \"68a706e9-72b8-4bb4-887a-0ea8044ae473\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:04.067263 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:04.067178 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:04.369943 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:04.369919 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8"] Sep 11 15:16:04.372254 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:16:04.372227 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a706e9_72b8_4bb4_887a_0ea8044ae473.slice/crio-bd92dee368f1a3cf4d49d7deba81c8b1c9dd5cc14d4c99270735bfadda06cf80 WatchSource:0}: Error finding container bd92dee368f1a3cf4d49d7deba81c8b1c9dd5cc14d4c99270735bfadda06cf80: Status 404 returned error can't find the container with id bd92dee368f1a3cf4d49d7deba81c8b1c9dd5cc14d4c99270735bfadda06cf80 Sep 11 15:16:04.470661 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:04.470632 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" event={"ID":"68a706e9-72b8-4bb4-887a-0ea8044ae473","Type":"ContainerStarted","Data":"bd92dee368f1a3cf4d49d7deba81c8b1c9dd5cc14d4c99270735bfadda06cf80"} Sep 11 15:16:07.486937 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:07.486902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" event={"ID":"68a706e9-72b8-4bb4-887a-0ea8044ae473","Type":"ContainerStarted","Data":"7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9"} Sep 11 15:16:07.487358 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:07.487022 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:07.512241 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:07.512199 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" podStartSLOduration=2.430720799 podStartE2EDuration="4.512184458s" totalImagesPullingTime="2.081463659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:16:07.510909312 +0000 UTC m=+449.556234293" watchObservedRunningTime="2026-09-11 15:16:07.512184458 +0000 UTC m=+449.557509437" Sep 11 15:16:15.701503 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.701467 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8"] Sep 11 15:16:15.701948 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.701723 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" podUID="68a706e9-72b8-4bb4-887a-0ea8044ae473" containerName="manager" containerID="cri-o://7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9" gracePeriod=2 Sep 11 15:16:15.703815 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.703766 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:15.743812 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.743768 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8"] Sep 11 15:16:15.924134 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.924103 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf"] Sep 11 15:16:15.924475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.924461 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68a706e9-72b8-4bb4-887a-0ea8044ae473" containerName="manager" Sep 11 15:16:15.924524 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.924477 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="68a706e9-72b8-4bb4-887a-0ea8044ae473" containerName="manager" Sep 11 15:16:15.924561 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.924530 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="68a706e9-72b8-4bb4-887a-0ea8044ae473" containerName="manager" Sep 11 15:16:15.927577 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.927552 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf" Sep 11 15:16:15.955269 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.955222 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:15.990909 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.990884 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5f5q\" (UniqueName: \"kubernetes.io/projected/68a706e9-72b8-4bb4-887a-0ea8044ae473-kube-api-access-v5f5q\") pod \"68a706e9-72b8-4bb4-887a-0ea8044ae473\" (UID: \"68a706e9-72b8-4bb4-887a-0ea8044ae473\") " Sep 11 15:16:15.991114 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.991094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbrf\" (UniqueName: \"kubernetes.io/projected/cdd28439-9ae9-4e7b-905b-9c32049e3514-kube-api-access-nhbrf\") pod \"limitador-operator-controller-manager-74495c69f8-vh8nf\" (UID: \"cdd28439-9ae9-4e7b-905b-9c32049e3514\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf" Sep 11 15:16:15.993093 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.993075 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a706e9-72b8-4bb4-887a-0ea8044ae473-kube-api-access-v5f5q" (OuterVolumeSpecName: "kube-api-access-v5f5q") pod "68a706e9-72b8-4bb4-887a-0ea8044ae473" (UID: "68a706e9-72b8-4bb4-887a-0ea8044ae473"). InnerVolumeSpecName "kube-api-access-v5f5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:16:15.999237 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:15.999216 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf"] Sep 11 15:16:16.046236 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:16:16.046209 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-84lb8\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="68a706e9-72b8-4bb4-887a-0ea8044ae473" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:16.048303 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:16:16.048283 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-84lb8\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="68a706e9-72b8-4bb4-887a-0ea8044ae473" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:16.092465 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.092443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbrf\" (UniqueName: \"kubernetes.io/projected/cdd28439-9ae9-4e7b-905b-9c32049e3514-kube-api-access-nhbrf\") pod \"limitador-operator-controller-manager-74495c69f8-vh8nf\" (UID: \"cdd28439-9ae9-4e7b-905b-9c32049e3514\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf" Sep 11 15:16:16.092557 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.092502 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5f5q\" (UniqueName: \"kubernetes.io/projected/68a706e9-72b8-4bb4-887a-0ea8044ae473-kube-api-access-v5f5q\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:16:16.106056 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.106037 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbrf\" (UniqueName: \"kubernetes.io/projected/cdd28439-9ae9-4e7b-905b-9c32049e3514-kube-api-access-nhbrf\") pod \"limitador-operator-controller-manager-74495c69f8-vh8nf\" (UID: \"cdd28439-9ae9-4e7b-905b-9c32049e3514\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf" Sep 11 15:16:16.250791 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.250721 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf" Sep 11 15:16:16.491848 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.491819 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68a706e9-72b8-4bb4-887a-0ea8044ae473" path="/var/lib/kubelet/pods/68a706e9-72b8-4bb4-887a-0ea8044ae473/volumes" Sep 11 15:16:16.528081 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.528026 2573 generic.go:356] "Generic (PLEG): container finished" podID="68a706e9-72b8-4bb4-887a-0ea8044ae473" containerID="7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9" exitCode=0 Sep 11 15:16:16.528081 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.528069 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-84lb8" Sep 11 15:16:16.528235 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.528129 2573 scope.go:117] "RemoveContainer" containerID="7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9" Sep 11 15:16:16.535959 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.535911 2573 scope.go:117] "RemoveContainer" containerID="7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9" Sep 11 15:16:16.536191 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:16:16.536174 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9\": container with ID starting with 7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9 not found: ID does not exist" containerID="7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9" Sep 11 15:16:16.536255 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.536206 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9"} err="failed to get container status \"7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9\": rpc error: code = NotFound desc = could not find container \"7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9\": container with ID starting with 7ca5f09f0ec288ac4b36d9ff39f28ba45db01947befc20a34eb1a1f2feed90b9 not found: ID does not exist" Sep 11 15:16:16.541592 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:16.541574 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf"] Sep 11 15:16:16.544109 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:16:16.544086 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd28439_9ae9_4e7b_905b_9c32049e3514.slice/crio-b4e852e3d931d2882c6bd206d51e748f4f3ca11bc6c85946c50a3e1745068a18 WatchSource:0}: Error finding container b4e852e3d931d2882c6bd206d51e748f4f3ca11bc6c85946c50a3e1745068a18: Status 404 returned error can't find the container with id b4e852e3d931d2882c6bd206d51e748f4f3ca11bc6c85946c50a3e1745068a18 Sep 11 15:16:17.535728 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:17.535689 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf" event={"ID":"cdd28439-9ae9-4e7b-905b-9c32049e3514","Type":"ContainerStarted","Data":"9a43ed9f0531f20941b0ceb13e506838411ab9f7aaec1c85103d7a9023a98d0f"} Sep 11 15:16:17.536102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:17.535748 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf" event={"ID":"cdd28439-9ae9-4e7b-905b-9c32049e3514","Type":"ContainerStarted","Data":"b4e852e3d931d2882c6bd206d51e748f4f3ca11bc6c85946c50a3e1745068a18"} Sep 11 15:16:17.536102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:17.535817 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf" Sep 11 15:16:17.601516 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:17.601472 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf" podStartSLOduration=2.6014586189999998 podStartE2EDuration="2.601458619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:16:17.597185235 +0000 UTC m=+459.642510214" watchObservedRunningTime="2026-09-11 15:16:17.601458619 +0000 UTC m=+459.646783598" Sep 11 15:16:28.541481 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:28.541453 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vh8nf" Sep 11 15:16:51.797218 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:51.797184 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-67mvc"] Sep 11 15:16:51.801857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:51.801838 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" Sep 11 15:16:51.804114 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:51.804095 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9k82x\"" Sep 11 15:16:51.806900 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:51.806878 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-67mvc"] Sep 11 15:16:51.869967 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:51.869932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7sz\" (UniqueName: \"kubernetes.io/projected/6780a981-e005-4635-ac55-49fefbc0cf1b-kube-api-access-6b7sz\") pod \"authorino-7bcfd8f9cd-67mvc\" (UID: \"6780a981-e005-4635-ac55-49fefbc0cf1b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" Sep 11 15:16:51.970558 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:51.970525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7sz\" (UniqueName: \"kubernetes.io/projected/6780a981-e005-4635-ac55-49fefbc0cf1b-kube-api-access-6b7sz\") pod \"authorino-7bcfd8f9cd-67mvc\" (UID: \"6780a981-e005-4635-ac55-49fefbc0cf1b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" Sep 11 15:16:51.977960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:51.977939 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7sz\" (UniqueName: \"kubernetes.io/projected/6780a981-e005-4635-ac55-49fefbc0cf1b-kube-api-access-6b7sz\") pod \"authorino-7bcfd8f9cd-67mvc\" (UID: \"6780a981-e005-4635-ac55-49fefbc0cf1b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" Sep 11 15:16:52.121968 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:52.121947 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" Sep 11 15:16:52.388178 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:52.388092 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-67mvc"] Sep 11 15:16:52.391986 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:16:52.391955 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6780a981_e005_4635_ac55_49fefbc0cf1b.slice/crio-c86779bd6b3e2d5ac5236946e071ed9b6b179d69f3291fdcb9b19153a541d96b WatchSource:0}: Error finding container c86779bd6b3e2d5ac5236946e071ed9b6b179d69f3291fdcb9b19153a541d96b: Status 404 returned error can't find the container with id c86779bd6b3e2d5ac5236946e071ed9b6b179d69f3291fdcb9b19153a541d96b Sep 11 15:16:52.692993 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:52.692923 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" event={"ID":"6780a981-e005-4635-ac55-49fefbc0cf1b","Type":"ContainerStarted","Data":"c86779bd6b3e2d5ac5236946e071ed9b6b179d69f3291fdcb9b19153a541d96b"} Sep 11 15:16:54.704319 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:54.704288 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" event={"ID":"6780a981-e005-4635-ac55-49fefbc0cf1b","Type":"ContainerStarted","Data":"7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0"} Sep 11 15:16:54.719280 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:54.719234 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" podStartSLOduration=1.796082656 podStartE2EDuration="3.719219407s" totalImagesPullingTime="1.923136751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:16:54.717882562 +0000 UTC m=+496.763207543" watchObservedRunningTime="2026-09-11 15:16:54.719219407 +0000 UTC m=+496.764544387" Sep 11 15:16:57.077423 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.077387 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-67mvc"] Sep 11 15:16:57.077804 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.077545 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" podUID="6780a981-e005-4635-ac55-49fefbc0cf1b" containerName="authorino" containerID="cri-o://7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0" gracePeriod=30 Sep 11 15:16:57.321661 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.321630 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" Sep 11 15:16:57.414443 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.414378 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b7sz\" (UniqueName: \"kubernetes.io/projected/6780a981-e005-4635-ac55-49fefbc0cf1b-kube-api-access-6b7sz\") pod \"6780a981-e005-4635-ac55-49fefbc0cf1b\" (UID: \"6780a981-e005-4635-ac55-49fefbc0cf1b\") " Sep 11 15:16:57.416428 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.416397 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6780a981-e005-4635-ac55-49fefbc0cf1b-kube-api-access-6b7sz" (OuterVolumeSpecName: "kube-api-access-6b7sz") pod "6780a981-e005-4635-ac55-49fefbc0cf1b" (UID: "6780a981-e005-4635-ac55-49fefbc0cf1b"). InnerVolumeSpecName "kube-api-access-6b7sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:16:57.515678 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.515648 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b7sz\" (UniqueName: \"kubernetes.io/projected/6780a981-e005-4635-ac55-49fefbc0cf1b-kube-api-access-6b7sz\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:16:57.718005 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.717943 2573 generic.go:356] "Generic (PLEG): container finished" podID="6780a981-e005-4635-ac55-49fefbc0cf1b" containerID="7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0" exitCode=0 Sep 11 15:16:57.718005 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.717998 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" Sep 11 15:16:57.718160 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.718041 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" event={"ID":"6780a981-e005-4635-ac55-49fefbc0cf1b","Type":"ContainerDied","Data":"7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0"} Sep 11 15:16:57.718160 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.718080 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-67mvc" event={"ID":"6780a981-e005-4635-ac55-49fefbc0cf1b","Type":"ContainerDied","Data":"c86779bd6b3e2d5ac5236946e071ed9b6b179d69f3291fdcb9b19153a541d96b"} Sep 11 15:16:57.718160 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.718109 2573 scope.go:117] "RemoveContainer" containerID="7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0" Sep 11 15:16:57.727333 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.727315 2573 scope.go:117] "RemoveContainer" containerID="7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0" Sep 11 15:16:57.727612 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:16:57.727588 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0\": container with ID starting with 7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0 not found: ID does not exist" containerID="7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0" Sep 11 15:16:57.727677 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.727624 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0"} err="failed to get container status \"7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0\": rpc error: code = NotFound desc = could not find container \"7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0\": container with ID starting with 7050dd043a6dbcc7404ecc74478d5dafe4f318b421a960380824aaafa30d33d0 not found: ID does not exist" Sep 11 15:16:57.738310 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.738286 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-67mvc"] Sep 11 15:16:57.740354 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:57.740335 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-67mvc"] Sep 11 15:16:58.492701 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:16:58.492671 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6780a981-e005-4635-ac55-49fefbc0cf1b" path="/var/lib/kubelet/pods/6780a981-e005-4635-ac55-49fefbc0cf1b/volumes" Sep 11 15:17:22.742917 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:22.742881 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-4dzww"] Sep 11 15:17:22.743260 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:22.743196 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6780a981-e005-4635-ac55-49fefbc0cf1b" containerName="authorino" Sep 11 15:17:22.743260 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:22.743208 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6780a981-e005-4635-ac55-49fefbc0cf1b" containerName="authorino" Sep 11 15:17:22.743330 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:22.743268 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6780a981-e005-4635-ac55-49fefbc0cf1b" containerName="authorino" Sep 11 15:17:22.747838 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:22.747824 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4dzww" Sep 11 15:17:22.750037 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:22.750019 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9k82x\"" Sep 11 15:17:22.753231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:22.753207 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-4dzww"] Sep 11 15:17:22.911914 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:22.911879 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmcvp\" (UniqueName: \"kubernetes.io/projected/f5f4f155-a90b-4c71-8659-62a00254c855-kube-api-access-mmcvp\") pod \"authorino-557f876f69-4dzww\" (UID: \"f5f4f155-a90b-4c71-8659-62a00254c855\") " pod="kuadrant-system/authorino-557f876f69-4dzww" Sep 11 15:17:22.975180 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:22.975150 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4dzww"] Sep 11 15:17:22.975414 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:17:22.975395 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mmcvp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-4dzww" podUID="f5f4f155-a90b-4c71-8659-62a00254c855" Sep 11 15:17:23.012897 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.012831 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmcvp\" (UniqueName: \"kubernetes.io/projected/f5f4f155-a90b-4c71-8659-62a00254c855-kube-api-access-mmcvp\") pod \"authorino-557f876f69-4dzww\" (UID: \"f5f4f155-a90b-4c71-8659-62a00254c855\") " pod="kuadrant-system/authorino-557f876f69-4dzww" Sep 11 15:17:23.022216 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.022196 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmcvp\" (UniqueName: \"kubernetes.io/projected/f5f4f155-a90b-4c71-8659-62a00254c855-kube-api-access-mmcvp\") pod \"authorino-557f876f69-4dzww\" (UID: \"f5f4f155-a90b-4c71-8659-62a00254c855\") " pod="kuadrant-system/authorino-557f876f69-4dzww" Sep 11 15:17:23.076036 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.076015 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78c55674cf-s248b"] Sep 11 15:17:23.079195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.079179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78c55674cf-s248b" Sep 11 15:17:23.081753 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.081736 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 15:17:23.089270 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.089248 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78c55674cf-s248b"] Sep 11 15:17:23.214246 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.214225 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e702820c-1190-4a53-8771-0f6ef20ecc6e-tls-cert\") pod \"authorino-78c55674cf-s248b\" (UID: \"e702820c-1190-4a53-8771-0f6ef20ecc6e\") " pod="kuadrant-system/authorino-78c55674cf-s248b" Sep 11 15:17:23.214343 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.214271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn996\" (UniqueName: \"kubernetes.io/projected/e702820c-1190-4a53-8771-0f6ef20ecc6e-kube-api-access-vn996\") pod \"authorino-78c55674cf-s248b\" (UID: \"e702820c-1190-4a53-8771-0f6ef20ecc6e\") " pod="kuadrant-system/authorino-78c55674cf-s248b" Sep 11 15:17:23.315324 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.315275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e702820c-1190-4a53-8771-0f6ef20ecc6e-tls-cert\") pod \"authorino-78c55674cf-s248b\" (UID: \"e702820c-1190-4a53-8771-0f6ef20ecc6e\") " pod="kuadrant-system/authorino-78c55674cf-s248b" Sep 11 15:17:23.315324 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.315320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn996\" (UniqueName: \"kubernetes.io/projected/e702820c-1190-4a53-8771-0f6ef20ecc6e-kube-api-access-vn996\") pod \"authorino-78c55674cf-s248b\" (UID: \"e702820c-1190-4a53-8771-0f6ef20ecc6e\") " pod="kuadrant-system/authorino-78c55674cf-s248b" Sep 11 15:17:23.317676 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.317651 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e702820c-1190-4a53-8771-0f6ef20ecc6e-tls-cert\") pod \"authorino-78c55674cf-s248b\" (UID: \"e702820c-1190-4a53-8771-0f6ef20ecc6e\") " pod="kuadrant-system/authorino-78c55674cf-s248b" Sep 11 15:17:23.322874 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.322856 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn996\" (UniqueName: \"kubernetes.io/projected/e702820c-1190-4a53-8771-0f6ef20ecc6e-kube-api-access-vn996\") pod \"authorino-78c55674cf-s248b\" (UID: \"e702820c-1190-4a53-8771-0f6ef20ecc6e\") " pod="kuadrant-system/authorino-78c55674cf-s248b" Sep 11 15:17:23.388348 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.388319 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78c55674cf-s248b" Sep 11 15:17:23.660470 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.660397 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78c55674cf-s248b"] Sep 11 15:17:23.662643 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:17:23.662616 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode702820c_1190_4a53_8771_0f6ef20ecc6e.slice/crio-7739463abd63791caf7ed51f77ffdcd704d69401cd3b35ceb8297a53f8c678c9 WatchSource:0}: Error finding container 7739463abd63791caf7ed51f77ffdcd704d69401cd3b35ceb8297a53f8c678c9: Status 404 returned error can't find the container with id 7739463abd63791caf7ed51f77ffdcd704d69401cd3b35ceb8297a53f8c678c9 Sep 11 15:17:23.832294 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.832268 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4dzww" Sep 11 15:17:23.832622 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.832292 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78c55674cf-s248b" event={"ID":"e702820c-1190-4a53-8771-0f6ef20ecc6e","Type":"ContainerStarted","Data":"7739463abd63791caf7ed51f77ffdcd704d69401cd3b35ceb8297a53f8c678c9"} Sep 11 15:17:23.840145 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:23.840126 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4dzww" Sep 11 15:17:24.022209 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:24.022156 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmcvp\" (UniqueName: \"kubernetes.io/projected/f5f4f155-a90b-4c71-8659-62a00254c855-kube-api-access-mmcvp\") pod \"f5f4f155-a90b-4c71-8659-62a00254c855\" (UID: \"f5f4f155-a90b-4c71-8659-62a00254c855\") " Sep 11 15:17:24.024346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:24.024310 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f4f155-a90b-4c71-8659-62a00254c855-kube-api-access-mmcvp" (OuterVolumeSpecName: "kube-api-access-mmcvp") pod "f5f4f155-a90b-4c71-8659-62a00254c855" (UID: "f5f4f155-a90b-4c71-8659-62a00254c855"). InnerVolumeSpecName "kube-api-access-mmcvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:17:24.122595 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:24.122573 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmcvp\" (UniqueName: \"kubernetes.io/projected/f5f4f155-a90b-4c71-8659-62a00254c855-kube-api-access-mmcvp\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:17:24.838691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:24.838652 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78c55674cf-s248b" event={"ID":"e702820c-1190-4a53-8771-0f6ef20ecc6e","Type":"ContainerStarted","Data":"def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010"} Sep 11 15:17:24.839149 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:24.838663 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4dzww" Sep 11 15:17:24.854870 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:24.854828 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78c55674cf-s248b" podStartSLOduration=1.000924526 podStartE2EDuration="1.85481349s" totalImagesPullingTime="853.888964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:17:24.853248505 +0000 UTC m=+526.898573498" watchObservedRunningTime="2026-09-11 15:17:24.85481349 +0000 UTC m=+526.900138469" Sep 11 15:17:24.877723 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:24.877695 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4dzww"] Sep 11 15:17:24.885572 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:24.885551 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-4dzww"] Sep 11 15:17:26.492521 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:17:26.492487 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f4f155-a90b-4c71-8659-62a00254c855" path="/var/lib/kubelet/pods/f5f4f155-a90b-4c71-8659-62a00254c855/volumes" Sep 11 15:18:08.083713 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.083682 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr"] Sep 11 15:18:08.087109 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.087090 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" Sep 11 15:18:08.091026 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.091001 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6tl98\"" Sep 11 15:18:08.091141 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.091050 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 15:18:08.108091 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.108070 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr"] Sep 11 15:18:08.134726 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.134695 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba070303-d713-4a08-87d2-473d485c4c21-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-x9hrr\" (UID: \"ba070303-d713-4a08-87d2-473d485c4c21\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" Sep 11 15:18:08.134872 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.134763 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8bjw\" (UniqueName: \"kubernetes.io/projected/ba070303-d713-4a08-87d2-473d485c4c21-kube-api-access-w8bjw\") pod \"payload-pre-processing-59fbc47bf8-x9hrr\" (UID: \"ba070303-d713-4a08-87d2-473d485c4c21\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" Sep 11 15:18:08.235949 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.235916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba070303-d713-4a08-87d2-473d485c4c21-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-x9hrr\" (UID: \"ba070303-d713-4a08-87d2-473d485c4c21\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" Sep 11 15:18:08.236113 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.235963 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8bjw\" (UniqueName: \"kubernetes.io/projected/ba070303-d713-4a08-87d2-473d485c4c21-kube-api-access-w8bjw\") pod \"payload-pre-processing-59fbc47bf8-x9hrr\" (UID: \"ba070303-d713-4a08-87d2-473d485c4c21\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" Sep 11 15:18:08.236693 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.236663 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba070303-d713-4a08-87d2-473d485c4c21-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-x9hrr\" (UID: \"ba070303-d713-4a08-87d2-473d485c4c21\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" Sep 11 15:18:08.250821 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.250765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8bjw\" (UniqueName: \"kubernetes.io/projected/ba070303-d713-4a08-87d2-473d485c4c21-kube-api-access-w8bjw\") pod \"payload-pre-processing-59fbc47bf8-x9hrr\" (UID: \"ba070303-d713-4a08-87d2-473d485c4c21\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" Sep 11 15:18:08.397310 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.397229 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" Sep 11 15:18:08.674967 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:08.674937 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr"] Sep 11 15:18:08.676232 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:18:08.676204 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba070303_d713_4a08_87d2_473d485c4c21.slice/crio-ccc23653b06906ff5f35ec44d6085bcce7a1afa6695e67396d5c230c7881f715 WatchSource:0}: Error finding container ccc23653b06906ff5f35ec44d6085bcce7a1afa6695e67396d5c230c7881f715: Status 404 returned error can't find the container with id ccc23653b06906ff5f35ec44d6085bcce7a1afa6695e67396d5c230c7881f715 Sep 11 15:18:09.038637 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:09.038559 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" event={"ID":"ba070303-d713-4a08-87d2-473d485c4c21","Type":"ContainerStarted","Data":"ccc23653b06906ff5f35ec44d6085bcce7a1afa6695e67396d5c230c7881f715"} Sep 11 15:18:12.055730 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:12.055699 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" event={"ID":"ba070303-d713-4a08-87d2-473d485c4c21","Type":"ContainerStarted","Data":"89efe527148661536dd0e2cf9e874dfbb7e83032d789b94cba3fb1a00b738e6b"} Sep 11 15:18:12.056085 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:12.055826 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" Sep 11 15:18:12.072756 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:12.072710 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" podStartSLOduration=1.319423237 podStartE2EDuration="4.072697631s" totalImagesPullingTime="2.753274394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:18:12.070921449 +0000 UTC m=+574.116246429" watchObservedRunningTime="2026-09-11 15:18:12.072697631 +0000 UTC m=+574.118022611" Sep 11 15:18:23.062658 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:23.062627 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9hrr" Sep 11 15:18:38.402690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:38.402660 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:18:38.403197 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:38.403180 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:18:46.692410 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.692381 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg"] Sep 11 15:18:46.697950 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.697927 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.700292 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.700267 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 15:18:46.700292 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.700280 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 15:18:46.700983 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.700965 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 11 15:18:46.700983 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.700982 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gfk8t\"" Sep 11 15:18:46.713468 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.711719 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg"] Sep 11 15:18:46.821953 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.821926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df8022c-5528-4100-a802-22fb65dd6791-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.822069 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.821960 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.822069 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.821982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jm7r\" (UniqueName: \"kubernetes.io/projected/0df8022c-5528-4100-a802-22fb65dd6791-kube-api-access-7jm7r\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.822069 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.822056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.822184 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.822078 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.822184 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.822103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.923347 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.923318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df8022c-5528-4100-a802-22fb65dd6791-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.923347 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.923349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.923498 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.923372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jm7r\" (UniqueName: \"kubernetes.io/projected/0df8022c-5528-4100-a802-22fb65dd6791-kube-api-access-7jm7r\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.923498 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.923437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.923498 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.923472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.923679 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.923657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.924050 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.924023 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.924319 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.924058 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.924584 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.924553 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.925954 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.925933 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df8022c-5528-4100-a802-22fb65dd6791-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.926259 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.926239 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df8022c-5528-4100-a802-22fb65dd6791-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:46.931549 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:46.931529 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jm7r\" (UniqueName: \"kubernetes.io/projected/0df8022c-5528-4100-a802-22fb65dd6791-kube-api-access-7jm7r\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg\" (UID: \"0df8022c-5528-4100-a802-22fb65dd6791\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:47.010601 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:47.010549 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:47.286931 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:47.286896 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg"] Sep 11 15:18:47.289087 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:18:47.289058 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df8022c_5528_4100_a802_22fb65dd6791.slice/crio-c3c5d66d36ec976608a3e0678ed023bdd814f0f7aabc61fc7639d9bedb001ed1 WatchSource:0}: Error finding container c3c5d66d36ec976608a3e0678ed023bdd814f0f7aabc61fc7639d9bedb001ed1: Status 404 returned error can't find the container with id c3c5d66d36ec976608a3e0678ed023bdd814f0f7aabc61fc7639d9bedb001ed1 Sep 11 15:18:48.220292 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:48.220252 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" event={"ID":"0df8022c-5528-4100-a802-22fb65dd6791","Type":"ContainerStarted","Data":"c3c5d66d36ec976608a3e0678ed023bdd814f0f7aabc61fc7639d9bedb001ed1"} Sep 11 15:18:49.226063 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:49.226026 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" event={"ID":"0df8022c-5528-4100-a802-22fb65dd6791","Type":"ContainerStarted","Data":"b6ee8d5c8dce011e706b079abb0ac9be6e7949b11ddd5eabf44611550014687f"} Sep 11 15:18:49.244045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:49.243997 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" podStartSLOduration=2.190248518 podStartE2EDuration="3.243980125s" totalImagesPullingTime="1.053731607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:18:49.243219457 +0000 UTC m=+611.288544437" watchObservedRunningTime="2026-09-11 15:18:49.243980125 +0000 UTC m=+611.289305133" Sep 11 15:18:57.010763 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:57.010726 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:57.013231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:57.010812 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:57.023492 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:57.023473 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:18:57.273757 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:18:57.273686 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg" Sep 11 15:19:13.778849 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.778743 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5"] Sep 11 15:19:13.784852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.784828 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.787263 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.787236 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 15:19:13.794300 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.794275 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5"] Sep 11 15:19:13.835084 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.835057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce98d31-f0f0-491b-b92d-afe822e568b1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.835207 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.835088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.835207 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.835117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9l56\" (UniqueName: \"kubernetes.io/projected/cce98d31-f0f0-491b-b92d-afe822e568b1-kube-api-access-t9l56\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.835207 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.835182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.835207 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.835202 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.835338 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.835264 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.936281 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.936256 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.936413 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.936283 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.936470 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.936416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.936523 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.936469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce98d31-f0f0-491b-b92d-afe822e568b1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.936523 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.936494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.936631 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.936523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9l56\" (UniqueName: \"kubernetes.io/projected/cce98d31-f0f0-491b-b92d-afe822e568b1-kube-api-access-t9l56\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.936686 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.936637 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.936735 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.936682 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.936735 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.936722 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.938856 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.938835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cce98d31-f0f0-491b-b92d-afe822e568b1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.939130 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.939112 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce98d31-f0f0-491b-b92d-afe822e568b1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:13.947962 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:13.947944 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9l56\" (UniqueName: \"kubernetes.io/projected/cce98d31-f0f0-491b-b92d-afe822e568b1-kube-api-access-t9l56\") pod \"e2e-trlp-test-simulated-kserve-598466548d-56hg5\" (UID: \"cce98d31-f0f0-491b-b92d-afe822e568b1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:14.096506 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:14.096478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:14.366345 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:14.366276 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5"] Sep 11 15:19:14.369346 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:19:14.369322 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce98d31_f0f0_491b_b92d_afe822e568b1.slice/crio-013b784e12e8476ff13db05df54ce15a37e8892ec7885c9df1446d8b4113b4e7 WatchSource:0}: Error finding container 013b784e12e8476ff13db05df54ce15a37e8892ec7885c9df1446d8b4113b4e7: Status 404 returned error can't find the container with id 013b784e12e8476ff13db05df54ce15a37e8892ec7885c9df1446d8b4113b4e7 Sep 11 15:19:14.370614 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:14.370593 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:19:15.350916 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:15.350882 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" event={"ID":"cce98d31-f0f0-491b-b92d-afe822e568b1","Type":"ContainerStarted","Data":"d0cf2aef272384586c9e0786a1149c610797f9c5567c1f4b038f60ef23da9743"} Sep 11 15:19:15.351351 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:15.350938 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" event={"ID":"cce98d31-f0f0-491b-b92d-afe822e568b1","Type":"ContainerStarted","Data":"013b784e12e8476ff13db05df54ce15a37e8892ec7885c9df1446d8b4113b4e7"} Sep 11 15:19:15.369555 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:15.369509 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" podStartSLOduration=2.179468891 podStartE2EDuration="2.369494805s" totalImagesPullingTime="190.025914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:19:15.367392777 +0000 UTC m=+637.412717757" watchObservedRunningTime="2026-09-11 15:19:15.369494805 +0000 UTC m=+637.414819785" Sep 11 15:19:24.096657 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:24.096621 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:24.097197 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:24.096687 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:24.109430 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:24.109406 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:24.406460 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:24.406390 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-56hg5" Sep 11 15:19:32.980553 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:32.980517 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl"] Sep 11 15:19:32.984432 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:32.984410 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:32.986878 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:32.986851 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 11 15:19:32.994200 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:32.994173 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl"] Sep 11 15:19:33.082122 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.082090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.082220 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.082134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea3459b-80ac-4374-8df3-f2b610097ad6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.082220 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.082167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.082308 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.082218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt6gp\" (UniqueName: \"kubernetes.io/projected/8ea3459b-80ac-4374-8df3-f2b610097ad6-kube-api-access-mt6gp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.082308 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.082236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.082308 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.082249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.183039 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.183011 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea3459b-80ac-4374-8df3-f2b610097ad6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.183120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.183053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.183120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.183096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt6gp\" (UniqueName: \"kubernetes.io/projected/8ea3459b-80ac-4374-8df3-f2b610097ad6-kube-api-access-mt6gp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.183193 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.183121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.183193 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.183145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.183284 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.183201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.183462 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.183439 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.183517 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.183486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.183557 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.183515 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.185491 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.185468 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ea3459b-80ac-4374-8df3-f2b610097ad6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.185644 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.185626 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea3459b-80ac-4374-8df3-f2b610097ad6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.191112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.191091 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt6gp\" (UniqueName: \"kubernetes.io/projected/8ea3459b-80ac-4374-8df3-f2b610097ad6-kube-api-access-mt6gp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6zxpl\" (UID: \"8ea3459b-80ac-4374-8df3-f2b610097ad6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.296103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.296053 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:33.580051 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:33.579898 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl"] Sep 11 15:19:33.582603 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:19:33.582574 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea3459b_80ac_4374_8df3_f2b610097ad6.slice/crio-1e38780f7da452fb0e0d6ba3b668610a0f18c5f508df773d3d283f8219a3a62e WatchSource:0}: Error finding container 1e38780f7da452fb0e0d6ba3b668610a0f18c5f508df773d3d283f8219a3a62e: Status 404 returned error can't find the container with id 1e38780f7da452fb0e0d6ba3b668610a0f18c5f508df773d3d283f8219a3a62e Sep 11 15:19:34.440607 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:34.440574 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" event={"ID":"8ea3459b-80ac-4374-8df3-f2b610097ad6","Type":"ContainerStarted","Data":"1cbac335fc8fedfd1e57f8bbc3c2dc1656a85bfd5bd6797f40854703e6ebb8bd"} Sep 11 15:19:34.440984 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:34.440627 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" event={"ID":"8ea3459b-80ac-4374-8df3-f2b610097ad6","Type":"ContainerStarted","Data":"1e38780f7da452fb0e0d6ba3b668610a0f18c5f508df773d3d283f8219a3a62e"} Sep 11 15:19:34.458251 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:34.458207 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" podStartSLOduration=2.260223134 podStartE2EDuration="2.458191651s" totalImagesPullingTime="197.968517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:19:34.45703615 +0000 UTC m=+656.502361130" watchObservedRunningTime="2026-09-11 15:19:34.458191651 +0000 UTC m=+656.503516631" Sep 11 15:19:43.296849 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:43.296820 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:43.297317 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:43.296915 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:43.309377 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:43.309350 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:19:43.495252 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:19:43.495231 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6zxpl" Sep 11 15:21:11.364690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.364653 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78c55674cf-s248b"] Sep 11 15:21:11.365217 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.364854 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-78c55674cf-s248b" podUID="e702820c-1190-4a53-8771-0f6ef20ecc6e" containerName="authorino" containerID="cri-o://def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010" gracePeriod=30 Sep 11 15:21:11.616237 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.616185 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78c55674cf-s248b" Sep 11 15:21:11.763234 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.763206 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn996\" (UniqueName: \"kubernetes.io/projected/e702820c-1190-4a53-8771-0f6ef20ecc6e-kube-api-access-vn996\") pod \"e702820c-1190-4a53-8771-0f6ef20ecc6e\" (UID: \"e702820c-1190-4a53-8771-0f6ef20ecc6e\") " Sep 11 15:21:11.763380 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.763245 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e702820c-1190-4a53-8771-0f6ef20ecc6e-tls-cert\") pod \"e702820c-1190-4a53-8771-0f6ef20ecc6e\" (UID: \"e702820c-1190-4a53-8771-0f6ef20ecc6e\") " Sep 11 15:21:11.765279 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.765251 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e702820c-1190-4a53-8771-0f6ef20ecc6e-kube-api-access-vn996" (OuterVolumeSpecName: "kube-api-access-vn996") pod "e702820c-1190-4a53-8771-0f6ef20ecc6e" (UID: "e702820c-1190-4a53-8771-0f6ef20ecc6e"). InnerVolumeSpecName "kube-api-access-vn996". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:21:11.773246 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.773216 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e702820c-1190-4a53-8771-0f6ef20ecc6e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e702820c-1190-4a53-8771-0f6ef20ecc6e" (UID: "e702820c-1190-4a53-8771-0f6ef20ecc6e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:21:11.864004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.863963 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn996\" (UniqueName: \"kubernetes.io/projected/e702820c-1190-4a53-8771-0f6ef20ecc6e-kube-api-access-vn996\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:21:11.864004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.864000 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e702820c-1190-4a53-8771-0f6ef20ecc6e-tls-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:21:11.876888 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.876832 2573 generic.go:356] "Generic (PLEG): container finished" podID="e702820c-1190-4a53-8771-0f6ef20ecc6e" containerID="def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010" exitCode=0 Sep 11 15:21:11.876888 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.876877 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78c55674cf-s248b" Sep 11 15:21:11.877008 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.876918 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78c55674cf-s248b" event={"ID":"e702820c-1190-4a53-8771-0f6ef20ecc6e","Type":"ContainerDied","Data":"def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010"} Sep 11 15:21:11.877008 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.876961 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78c55674cf-s248b" event={"ID":"e702820c-1190-4a53-8771-0f6ef20ecc6e","Type":"ContainerDied","Data":"7739463abd63791caf7ed51f77ffdcd704d69401cd3b35ceb8297a53f8c678c9"} Sep 11 15:21:11.877008 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.876981 2573 scope.go:117] "RemoveContainer" containerID="def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010" Sep 11 15:21:11.885303 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.885285 2573 scope.go:117] "RemoveContainer" containerID="def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010" Sep 11 15:21:11.885640 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:21:11.885620 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010\": container with ID starting with def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010 not found: ID does not exist" containerID="def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010" Sep 11 15:21:11.885703 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.885651 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010"} err="failed to get container status \"def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010\": rpc error: code = NotFound desc = could not find container \"def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010\": container with ID starting with def09fb588261cf878ec49ece3ffed74fe23716420587be0be6bfcbe41e51010 not found: ID does not exist" Sep 11 15:21:11.899418 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.899394 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78c55674cf-s248b"] Sep 11 15:21:11.903542 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:11.903521 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78c55674cf-s248b"] Sep 11 15:21:12.493689 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:12.493657 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e702820c-1190-4a53-8771-0f6ef20ecc6e" path="/var/lib/kubelet/pods/e702820c-1190-4a53-8771-0f6ef20ecc6e/volumes" Sep 11 15:21:31.644678 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.644641 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7349-edf222"] Sep 11 15:21:31.645084 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.644995 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e702820c-1190-4a53-8771-0f6ef20ecc6e" containerName="authorino" Sep 11 15:21:31.645084 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.645006 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e702820c-1190-4a53-8771-0f6ef20ecc6e" containerName="authorino" Sep 11 15:21:31.645084 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.645070 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e702820c-1190-4a53-8771-0f6ef20ecc6e" containerName="authorino" Sep 11 15:21:31.649395 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.649378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7349-edf222" Sep 11 15:21:31.651922 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.651898 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6mbx9\"" Sep 11 15:21:31.653015 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.652988 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7349-edf222"] Sep 11 15:21:31.695565 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.695543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qq88\" (UniqueName: \"kubernetes.io/projected/1aabadb7-4fca-4e28-b649-40e12977a19b-kube-api-access-7qq88\") pod \"test-curl-7349-edf222\" (UID: \"1aabadb7-4fca-4e28-b649-40e12977a19b\") " pod="opendatahub/test-curl-7349-edf222" Sep 11 15:21:31.796366 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.796341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qq88\" (UniqueName: \"kubernetes.io/projected/1aabadb7-4fca-4e28-b649-40e12977a19b-kube-api-access-7qq88\") pod \"test-curl-7349-edf222\" (UID: \"1aabadb7-4fca-4e28-b649-40e12977a19b\") " pod="opendatahub/test-curl-7349-edf222" Sep 11 15:21:31.804164 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.804136 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qq88\" (UniqueName: \"kubernetes.io/projected/1aabadb7-4fca-4e28-b649-40e12977a19b-kube-api-access-7qq88\") pod \"test-curl-7349-edf222\" (UID: \"1aabadb7-4fca-4e28-b649-40e12977a19b\") " pod="opendatahub/test-curl-7349-edf222" Sep 11 15:21:31.965121 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:31.965045 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7349-edf222" Sep 11 15:21:32.236603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:32.236578 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7349-edf222"] Sep 11 15:21:32.238865 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:21:32.238838 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1aabadb7_4fca_4e28_b649_40e12977a19b.slice/crio-651a02c8d42e9ccc07b2d9d7c5a09f7c81565c12b4a4be9ebece5452151cb4dd WatchSource:0}: Error finding container 651a02c8d42e9ccc07b2d9d7c5a09f7c81565c12b4a4be9ebece5452151cb4dd: Status 404 returned error can't find the container with id 651a02c8d42e9ccc07b2d9d7c5a09f7c81565c12b4a4be9ebece5452151cb4dd Sep 11 15:21:32.975138 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:32.975102 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7349-edf222" event={"ID":"1aabadb7-4fca-4e28-b649-40e12977a19b","Type":"ContainerStarted","Data":"651a02c8d42e9ccc07b2d9d7c5a09f7c81565c12b4a4be9ebece5452151cb4dd"} Sep 11 15:21:34.987171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:34.987134 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7349-edf222" event={"ID":"1aabadb7-4fca-4e28-b649-40e12977a19b","Type":"ContainerStarted","Data":"2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd"} Sep 11 15:21:35.001202 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.001151 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7349-edf222" podStartSLOduration=1.920733599 podStartE2EDuration="4.001131689s" totalImagesPullingTime="2.08039809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:35.001121052 +0000 UTC m=+777.046446057" watchObservedRunningTime="2026-09-11 15:21:35.001131689 +0000 UTC m=+777.046456696" Sep 11 15:21:35.438347 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.438318 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc"] Sep 11 15:21:35.441638 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.441622 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:21:35.443950 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.443930 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-778341-dockercfg-xngqm\"" Sep 11 15:21:35.444026 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.444005 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-778341\"" Sep 11 15:21:35.453884 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.453863 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc"] Sep 11 15:21:35.496786 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.496745 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7"] Sep 11 15:21:35.500041 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.500027 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:21:35.512125 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.512104 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7"] Sep 11 15:21:35.532485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.532449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrc5w\" (UniqueName: \"kubernetes.io/projected/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-kube-api-access-jrc5w\") pod \"payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc\" (UID: \"17e643a4-8ab8-4621-ace9-3bbc56c0fab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:21:35.532674 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.532633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc\" (UID: \"17e643a4-8ab8-4621-ace9-3bbc56c0fab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:21:35.532816 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.532767 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l4x7\" (UniqueName: \"kubernetes.io/projected/42c42ed8-71fe-492e-80c6-97a5b2c864ad-kube-api-access-7l4x7\") pod \"payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7\" (UID: \"42c42ed8-71fe-492e-80c6-97a5b2c864ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:21:35.532883 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.532837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c42ed8-71fe-492e-80c6-97a5b2c864ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7\" (UID: \"42c42ed8-71fe-492e-80c6-97a5b2c864ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:21:35.633456 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.633428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrc5w\" (UniqueName: \"kubernetes.io/projected/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-kube-api-access-jrc5w\") pod \"payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc\" (UID: \"17e643a4-8ab8-4621-ace9-3bbc56c0fab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:21:35.633590 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.633468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc\" (UID: \"17e643a4-8ab8-4621-ace9-3bbc56c0fab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:21:35.633590 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.633515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4x7\" (UniqueName: \"kubernetes.io/projected/42c42ed8-71fe-492e-80c6-97a5b2c864ad-kube-api-access-7l4x7\") pod \"payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7\" (UID: \"42c42ed8-71fe-492e-80c6-97a5b2c864ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:21:35.633590 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.633541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c42ed8-71fe-492e-80c6-97a5b2c864ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7\" (UID: \"42c42ed8-71fe-492e-80c6-97a5b2c864ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:21:35.634454 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.634431 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc\" (UID: \"17e643a4-8ab8-4621-ace9-3bbc56c0fab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:21:35.634513 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.634431 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c42ed8-71fe-492e-80c6-97a5b2c864ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7\" (UID: \"42c42ed8-71fe-492e-80c6-97a5b2c864ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:21:35.642767 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.642740 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4x7\" (UniqueName: \"kubernetes.io/projected/42c42ed8-71fe-492e-80c6-97a5b2c864ad-kube-api-access-7l4x7\") pod \"payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7\" (UID: \"42c42ed8-71fe-492e-80c6-97a5b2c864ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:21:35.643062 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.643045 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrc5w\" (UniqueName: \"kubernetes.io/projected/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-kube-api-access-jrc5w\") pod \"payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc\" (UID: \"17e643a4-8ab8-4621-ace9-3bbc56c0fab8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:21:35.647930 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.647909 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7349-edf222"] Sep 11 15:21:35.653728 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.653708 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7349-edf222"] Sep 11 15:21:35.751796 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.751723 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:21:35.783238 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.783210 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7349-d5dfc8"] Sep 11 15:21:35.783621 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.783605 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aabadb7-4fca-4e28-b649-40e12977a19b" containerName="test-curl-7349-edf222" Sep 11 15:21:35.783694 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.783623 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aabadb7-4fca-4e28-b649-40e12977a19b" containerName="test-curl-7349-edf222" Sep 11 15:21:35.783758 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.783697 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aabadb7-4fca-4e28-b649-40e12977a19b" containerName="test-curl-7349-edf222" Sep 11 15:21:35.786844 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.786824 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7349-d5dfc8" Sep 11 15:21:35.794067 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.794022 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7349-d5dfc8"] Sep 11 15:21:35.810173 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.810150 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:21:35.836142 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.836066 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78g25\" (UniqueName: \"kubernetes.io/projected/94c5f04b-3e90-4899-b257-f4dd8ffe7be1-kube-api-access-78g25\") pod \"test-curl-7349-d5dfc8\" (UID: \"94c5f04b-3e90-4899-b257-f4dd8ffe7be1\") " pod="opendatahub/test-curl-7349-d5dfc8" Sep 11 15:21:35.937054 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.937026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78g25\" (UniqueName: \"kubernetes.io/projected/94c5f04b-3e90-4899-b257-f4dd8ffe7be1-kube-api-access-78g25\") pod \"test-curl-7349-d5dfc8\" (UID: \"94c5f04b-3e90-4899-b257-f4dd8ffe7be1\") " pod="opendatahub/test-curl-7349-d5dfc8" Sep 11 15:21:35.944638 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:35.944619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78g25\" (UniqueName: \"kubernetes.io/projected/94c5f04b-3e90-4899-b257-f4dd8ffe7be1-kube-api-access-78g25\") pod \"test-curl-7349-d5dfc8\" (UID: \"94c5f04b-3e90-4899-b257-f4dd8ffe7be1\") " pod="opendatahub/test-curl-7349-d5dfc8" Sep 11 15:21:35.997377 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:21:35.993872 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7349-edf222" container="test-curl-7349-edf222" Sep 11 15:21:36.025132 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:36.025066 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc"] Sep 11 15:21:36.027391 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:21:36.027369 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e643a4_8ab8_4621_ace9_3bbc56c0fab8.slice/crio-f9e0303d6b328eea2f415e3f072eb1aaf37a136c6bbf3025eae2675c0be7f2d3 WatchSource:0}: Error finding container f9e0303d6b328eea2f415e3f072eb1aaf37a136c6bbf3025eae2675c0be7f2d3: Status 404 returned error can't find the container with id f9e0303d6b328eea2f415e3f072eb1aaf37a136c6bbf3025eae2675c0be7f2d3 Sep 11 15:21:36.088043 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:36.088017 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7"] Sep 11 15:21:36.089815 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:21:36.089787 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c42ed8_71fe_492e_80c6_97a5b2c864ad.slice/crio-b7c1103f47c019635b6e95275cd8156b2bebea63f649288fabbba03e2dc5a3ff WatchSource:0}: Error finding container b7c1103f47c019635b6e95275cd8156b2bebea63f649288fabbba03e2dc5a3ff: Status 404 returned error can't find the container with id b7c1103f47c019635b6e95275cd8156b2bebea63f649288fabbba03e2dc5a3ff Sep 11 15:21:36.097853 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:36.097833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7349-d5dfc8" Sep 11 15:21:36.373541 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:36.373509 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7349-d5dfc8"] Sep 11 15:21:36.376342 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:21:36.376316 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94c5f04b_3e90_4899_b257_f4dd8ffe7be1.slice/crio-a6ffc793ecb258ef88a28bea1a9c4b9c5475cf343a0352305c5bc88c21042b6b WatchSource:0}: Error finding container a6ffc793ecb258ef88a28bea1a9c4b9c5475cf343a0352305c5bc88c21042b6b: Status 404 returned error can't find the container with id a6ffc793ecb258ef88a28bea1a9c4b9c5475cf343a0352305c5bc88c21042b6b Sep 11 15:21:37.000901 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.000859 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7349-d5dfc8" event={"ID":"94c5f04b-3e90-4899-b257-f4dd8ffe7be1","Type":"ContainerStarted","Data":"38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18"} Sep 11 15:21:37.001313 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.000908 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7349-d5dfc8" event={"ID":"94c5f04b-3e90-4899-b257-f4dd8ffe7be1","Type":"ContainerStarted","Data":"a6ffc793ecb258ef88a28bea1a9c4b9c5475cf343a0352305c5bc88c21042b6b"} Sep 11 15:21:37.002585 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.002559 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" event={"ID":"42c42ed8-71fe-492e-80c6-97a5b2c864ad","Type":"ContainerStarted","Data":"afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44"} Sep 11 15:21:37.002725 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.002606 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" event={"ID":"42c42ed8-71fe-492e-80c6-97a5b2c864ad","Type":"ContainerStarted","Data":"b7c1103f47c019635b6e95275cd8156b2bebea63f649288fabbba03e2dc5a3ff"} Sep 11 15:21:37.002725 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.002677 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:21:37.004144 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.004115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" event={"ID":"17e643a4-8ab8-4621-ace9-3bbc56c0fab8","Type":"ContainerStarted","Data":"c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277"} Sep 11 15:21:37.004249 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.004195 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" event={"ID":"17e643a4-8ab8-4621-ace9-3bbc56c0fab8","Type":"ContainerStarted","Data":"f9e0303d6b328eea2f415e3f072eb1aaf37a136c6bbf3025eae2675c0be7f2d3"} Sep 11 15:21:37.004249 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.004140 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7349-edf222" podUID="1aabadb7-4fca-4e28-b649-40e12977a19b" containerName="test-curl-7349-edf222" containerID="cri-o://2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd" gracePeriod=2 Sep 11 15:21:37.016125 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.016088 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7349-d5dfc8" podStartSLOduration=2.016075739 podStartE2EDuration="2.016075739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:37.014626757 +0000 UTC m=+779.059951738" watchObservedRunningTime="2026-09-11 15:21:37.016075739 +0000 UTC m=+779.061400720" Sep 11 15:21:37.032537 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.032493 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" podStartSLOduration=2.032481148 podStartE2EDuration="2.032481148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:37.030721738 +0000 UTC m=+779.076046721" watchObservedRunningTime="2026-09-11 15:21:37.032481148 +0000 UTC m=+779.077806129" Sep 11 15:21:37.046180 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.046136 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" podStartSLOduration=2.046119777 podStartE2EDuration="2.046119777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:37.045475074 +0000 UTC m=+779.090800055" watchObservedRunningTime="2026-09-11 15:21:37.046119777 +0000 UTC m=+779.091444758" Sep 11 15:21:37.647658 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.647626 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7349-d5dfc8"] Sep 11 15:21:37.653499 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:37.653475 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7349-d5dfc8"] Sep 11 15:21:38.009203 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:21:38.009169 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7349-d5dfc8" container="test-curl-7349-d5dfc8" Sep 11 15:21:38.009553 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:38.009303 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:21:39.013916 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:39.013883 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7349-d5dfc8" podUID="94c5f04b-3e90-4899-b257-f4dd8ffe7be1" containerName="test-curl-7349-d5dfc8" containerID="cri-o://38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18" gracePeriod=2 Sep 11 15:21:39.183225 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:39.183195 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7349-edf222" Sep 11 15:21:39.269604 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:39.269542 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qq88\" (UniqueName: \"kubernetes.io/projected/1aabadb7-4fca-4e28-b649-40e12977a19b-kube-api-access-7qq88\") pod \"1aabadb7-4fca-4e28-b649-40e12977a19b\" (UID: \"1aabadb7-4fca-4e28-b649-40e12977a19b\") " Sep 11 15:21:39.271880 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:39.271854 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aabadb7-4fca-4e28-b649-40e12977a19b-kube-api-access-7qq88" (OuterVolumeSpecName: "kube-api-access-7qq88") pod "1aabadb7-4fca-4e28-b649-40e12977a19b" (UID: "1aabadb7-4fca-4e28-b649-40e12977a19b"). InnerVolumeSpecName "kube-api-access-7qq88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:21:39.370734 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:39.370710 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qq88\" (UniqueName: \"kubernetes.io/projected/1aabadb7-4fca-4e28-b649-40e12977a19b-kube-api-access-7qq88\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:21:40.020139 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:40.020105 2573 generic.go:356] "Generic (PLEG): container finished" podID="1aabadb7-4fca-4e28-b649-40e12977a19b" containerID="2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd" exitCode=137 Sep 11 15:21:40.020553 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:40.020148 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7349-edf222" Sep 11 15:21:40.020553 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:40.020166 2573 scope.go:117] "RemoveContainer" containerID="2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd" Sep 11 15:21:40.029596 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:40.029579 2573 scope.go:117] "RemoveContainer" containerID="2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd" Sep 11 15:21:40.029913 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:21:40.029889 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd\": container with ID starting with 2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd not found: ID does not exist" containerID="2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd" Sep 11 15:21:40.029981 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:40.029924 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd"} err="failed to get container status \"2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd\": rpc error: code = NotFound desc = could not find container \"2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd\": container with ID starting with 2845d63fd2b7fc07b341a1caeffc0d8095685a440cf4b7677bb10c439a421afd not found: ID does not exist" Sep 11 15:21:40.030829 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:21:40.030801 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7349-edf222\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="1aabadb7-4fca-4e28-b649-40e12977a19b" pod="opendatahub/test-curl-7349-edf222" Sep 11 15:21:40.492618 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:40.492587 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aabadb7-4fca-4e28-b649-40e12977a19b" path="/var/lib/kubelet/pods/1aabadb7-4fca-4e28-b649-40e12977a19b/volumes" Sep 11 15:21:41.180751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.180728 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7349-d5dfc8" Sep 11 15:21:41.286024 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.285968 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78g25\" (UniqueName: \"kubernetes.io/projected/94c5f04b-3e90-4899-b257-f4dd8ffe7be1-kube-api-access-78g25\") pod \"94c5f04b-3e90-4899-b257-f4dd8ffe7be1\" (UID: \"94c5f04b-3e90-4899-b257-f4dd8ffe7be1\") " Sep 11 15:21:41.288029 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.287999 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c5f04b-3e90-4899-b257-f4dd8ffe7be1-kube-api-access-78g25" (OuterVolumeSpecName: "kube-api-access-78g25") pod "94c5f04b-3e90-4899-b257-f4dd8ffe7be1" (UID: "94c5f04b-3e90-4899-b257-f4dd8ffe7be1"). InnerVolumeSpecName "kube-api-access-78g25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:21:41.386881 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.386856 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78g25\" (UniqueName: \"kubernetes.io/projected/94c5f04b-3e90-4899-b257-f4dd8ffe7be1-kube-api-access-78g25\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:21:41.786223 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.786193 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd"] Sep 11 15:21:41.786577 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.786565 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c5f04b-3e90-4899-b257-f4dd8ffe7be1" containerName="test-curl-7349-d5dfc8" Sep 11 15:21:41.786619 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.786579 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c5f04b-3e90-4899-b257-f4dd8ffe7be1" containerName="test-curl-7349-d5dfc8" Sep 11 15:21:41.786654 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.786635 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94c5f04b-3e90-4899-b257-f4dd8ffe7be1" containerName="test-curl-7349-d5dfc8" Sep 11 15:21:41.789621 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.789604 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:21:41.792291 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.792263 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6e6b7dd7\"" Sep 11 15:21:41.799722 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.799695 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd"] Sep 11 15:21:41.801864 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.801840 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd"] Sep 11 15:21:41.805786 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.805756 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:21:41.808109 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.808091 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6e6b7dd7-dockercfg-bb4zz\"" Sep 11 15:21:41.816893 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.816873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd"] Sep 11 15:21:41.891141 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.891109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd\" (UID: \"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c\") " pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:21:41.891253 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.891152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg7f8\" (UniqueName: \"kubernetes.io/projected/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-kube-api-access-vg7f8\") pod \"payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd\" (UID: \"4fe688c0-17dd-47a3-bf91-ddeb00a40a90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:21:41.891253 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.891229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlqtv\" (UniqueName: \"kubernetes.io/projected/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-kube-api-access-jlqtv\") pod \"payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd\" (UID: \"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c\") " pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:21:41.891331 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.891267 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd\" (UID: \"4fe688c0-17dd-47a3-bf91-ddeb00a40a90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:21:41.992368 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.992337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd\" (UID: \"4fe688c0-17dd-47a3-bf91-ddeb00a40a90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:21:41.992485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.992390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd\" (UID: \"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c\") " pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:21:41.992485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.992417 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg7f8\" (UniqueName: \"kubernetes.io/projected/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-kube-api-access-vg7f8\") pod \"payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd\" (UID: \"4fe688c0-17dd-47a3-bf91-ddeb00a40a90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:21:41.992485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.992456 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlqtv\" (UniqueName: \"kubernetes.io/projected/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-kube-api-access-jlqtv\") pod \"payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd\" (UID: \"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c\") " pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:21:41.993107 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.993089 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd\" (UID: \"4fe688c0-17dd-47a3-bf91-ddeb00a40a90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:21:41.993147 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.993126 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd\" (UID: \"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c\") " pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:21:41.999984 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:41.999954 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlqtv\" (UniqueName: \"kubernetes.io/projected/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-kube-api-access-jlqtv\") pod \"payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd\" (UID: \"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c\") " pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:21:42.000105 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.000051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg7f8\" (UniqueName: \"kubernetes.io/projected/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-kube-api-access-vg7f8\") pod \"payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd\" (UID: \"4fe688c0-17dd-47a3-bf91-ddeb00a40a90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:21:42.032449 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.032428 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7349-d5dfc8" Sep 11 15:21:42.032573 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.032453 2573 scope.go:117] "RemoveContainer" containerID="38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18" Sep 11 15:21:42.032573 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.032390 2573 generic.go:356] "Generic (PLEG): container finished" podID="94c5f04b-3e90-4899-b257-f4dd8ffe7be1" containerID="38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18" exitCode=137 Sep 11 15:21:42.041584 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.041565 2573 scope.go:117] "RemoveContainer" containerID="38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18" Sep 11 15:21:42.041858 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:21:42.041840 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18\": container with ID starting with 38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18 not found: ID does not exist" containerID="38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18" Sep 11 15:21:42.041937 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.041869 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18"} err="failed to get container status \"38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18\": rpc error: code = NotFound desc = could not find container \"38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18\": container with ID starting with 38ac0b14e3b1dc32c7347a6cbd30c57aa6d23e1455ad0be5e84cfc9a4ad49d18 not found: ID does not exist" Sep 11 15:21:42.043145 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:21:42.043123 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7349-d5dfc8\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="94c5f04b-3e90-4899-b257-f4dd8ffe7be1" pod="opendatahub/test-curl-7349-d5dfc8" Sep 11 15:21:42.101740 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.101716 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:21:42.117698 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.117676 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:21:42.372015 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.371984 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd"] Sep 11 15:21:42.374520 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:21:42.374493 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe688c0_17dd_47a3_bf91_ddeb00a40a90.slice/crio-ab7653d564e8fde35e916ca0226a3873524df6eb1dee1501110d52b5ec8e9019 WatchSource:0}: Error finding container ab7653d564e8fde35e916ca0226a3873524df6eb1dee1501110d52b5ec8e9019: Status 404 returned error can't find the container with id ab7653d564e8fde35e916ca0226a3873524df6eb1dee1501110d52b5ec8e9019 Sep 11 15:21:42.395004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.394969 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd"] Sep 11 15:21:42.397094 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:21:42.397068 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a7b401_9cf0_4853_b4f8_cf3766e35e8c.slice/crio-bf9aa4a72e65d1f255f5fbde2fb79a3a0477581f49cd676131b5b73b4d39a5de WatchSource:0}: Error finding container bf9aa4a72e65d1f255f5fbde2fb79a3a0477581f49cd676131b5b73b4d39a5de: Status 404 returned error can't find the container with id bf9aa4a72e65d1f255f5fbde2fb79a3a0477581f49cd676131b5b73b4d39a5de Sep 11 15:21:42.495821 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:42.495752 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94c5f04b-3e90-4899-b257-f4dd8ffe7be1" path="/var/lib/kubelet/pods/94c5f04b-3e90-4899-b257-f4dd8ffe7be1/volumes" Sep 11 15:21:43.039197 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:43.039160 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" event={"ID":"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c","Type":"ContainerStarted","Data":"96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737"} Sep 11 15:21:43.039371 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:43.039233 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" event={"ID":"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c","Type":"ContainerStarted","Data":"bf9aa4a72e65d1f255f5fbde2fb79a3a0477581f49cd676131b5b73b4d39a5de"} Sep 11 15:21:43.039371 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:43.039287 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:21:43.040764 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:43.040738 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" event={"ID":"4fe688c0-17dd-47a3-bf91-ddeb00a40a90","Type":"ContainerStarted","Data":"fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b"} Sep 11 15:21:43.040902 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:43.040792 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" event={"ID":"4fe688c0-17dd-47a3-bf91-ddeb00a40a90","Type":"ContainerStarted","Data":"ab7653d564e8fde35e916ca0226a3873524df6eb1dee1501110d52b5ec8e9019"} Sep 11 15:21:43.040902 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:43.040833 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:21:43.060144 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:43.060082 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" podStartSLOduration=2.06006908 podStartE2EDuration="2.06006908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:43.057727096 +0000 UTC m=+785.103052087" watchObservedRunningTime="2026-09-11 15:21:43.06006908 +0000 UTC m=+785.105394060" Sep 11 15:21:43.078662 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:43.078624 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" podStartSLOduration=2.078611836 podStartE2EDuration="2.078611836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:43.076541627 +0000 UTC m=+785.121866618" watchObservedRunningTime="2026-09-11 15:21:43.078611836 +0000 UTC m=+785.123936862" Sep 11 15:21:47.177951 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.177916 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q"] Sep 11 15:21:47.181592 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.181575 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:21:47.183996 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.183970 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-34c71e-dockercfg-5k9st\"" Sep 11 15:21:47.184136 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.184001 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-34c71e\"" Sep 11 15:21:47.190512 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.190487 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q"] Sep 11 15:21:47.205478 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.205451 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg"] Sep 11 15:21:47.208862 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.208847 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:21:47.215630 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.215605 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg"] Sep 11 15:21:47.238825 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.238790 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q\" (UID: \"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:21:47.238938 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.238855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdhl\" (UniqueName: \"kubernetes.io/projected/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-kube-api-access-nmdhl\") pod \"payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q\" (UID: \"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:21:47.339298 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.339273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q\" (UID: \"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:21:47.339419 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.339323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdhl\" (UniqueName: \"kubernetes.io/projected/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-kube-api-access-nmdhl\") pod \"payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q\" (UID: \"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:21:47.339419 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.339355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab651a9-7564-4d22-9d39-462c40b179d2-plugins-config-volume\") pod \"payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg\" (UID: \"9ab651a9-7564-4d22-9d39-462c40b179d2\") " pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:21:47.339419 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.339378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjxb\" (UniqueName: \"kubernetes.io/projected/9ab651a9-7564-4d22-9d39-462c40b179d2-kube-api-access-bjjxb\") pod \"payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg\" (UID: \"9ab651a9-7564-4d22-9d39-462c40b179d2\") " pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:21:47.339990 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.339971 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q\" (UID: \"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:21:47.347073 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.347054 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdhl\" (UniqueName: \"kubernetes.io/projected/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-kube-api-access-nmdhl\") pod \"payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q\" (UID: \"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:21:47.440486 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.440415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab651a9-7564-4d22-9d39-462c40b179d2-plugins-config-volume\") pod \"payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg\" (UID: \"9ab651a9-7564-4d22-9d39-462c40b179d2\") " pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:21:47.440486 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.440462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjxb\" (UniqueName: \"kubernetes.io/projected/9ab651a9-7564-4d22-9d39-462c40b179d2-kube-api-access-bjjxb\") pod \"payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg\" (UID: \"9ab651a9-7564-4d22-9d39-462c40b179d2\") " pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:21:47.441293 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.441269 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab651a9-7564-4d22-9d39-462c40b179d2-plugins-config-volume\") pod \"payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg\" (UID: \"9ab651a9-7564-4d22-9d39-462c40b179d2\") " pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:21:47.450591 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.450563 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjxb\" (UniqueName: \"kubernetes.io/projected/9ab651a9-7564-4d22-9d39-462c40b179d2-kube-api-access-bjjxb\") pod \"payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg\" (UID: \"9ab651a9-7564-4d22-9d39-462c40b179d2\") " pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:21:47.493757 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.493731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:21:47.520329 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.520304 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:21:47.767902 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.767874 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q"] Sep 11 15:21:47.769689 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:21:47.769667 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cb2f2f_b3a0_429d_91bb_dbdfa30fd9e4.slice/crio-a444403e0dc64052cc81b82507827a24a02c1215d702869d2a7dc08037a54fbb WatchSource:0}: Error finding container a444403e0dc64052cc81b82507827a24a02c1215d702869d2a7dc08037a54fbb: Status 404 returned error can't find the container with id a444403e0dc64052cc81b82507827a24a02c1215d702869d2a7dc08037a54fbb Sep 11 15:21:47.810589 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:47.810565 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg"] Sep 11 15:21:47.817026 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:21:47.817000 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab651a9_7564_4d22_9d39_462c40b179d2.slice/crio-fa403b4d191651f0ccb6d35623e902f616b89475dacc48f73274c2c6a7fa220b WatchSource:0}: Error finding container fa403b4d191651f0ccb6d35623e902f616b89475dacc48f73274c2c6a7fa220b: Status 404 returned error can't find the container with id fa403b4d191651f0ccb6d35623e902f616b89475dacc48f73274c2c6a7fa220b Sep 11 15:21:48.011261 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:48.011231 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:21:48.075306 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:48.075270 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" event={"ID":"9ab651a9-7564-4d22-9d39-462c40b179d2","Type":"ContainerStarted","Data":"56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908"} Sep 11 15:21:48.075494 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:48.075387 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:21:48.075494 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:48.075411 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" event={"ID":"9ab651a9-7564-4d22-9d39-462c40b179d2","Type":"ContainerStarted","Data":"fa403b4d191651f0ccb6d35623e902f616b89475dacc48f73274c2c6a7fa220b"} Sep 11 15:21:48.077286 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:48.077262 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" event={"ID":"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4","Type":"ContainerStarted","Data":"bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5"} Sep 11 15:21:48.077397 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:48.077310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" event={"ID":"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4","Type":"ContainerStarted","Data":"a444403e0dc64052cc81b82507827a24a02c1215d702869d2a7dc08037a54fbb"} Sep 11 15:21:48.077397 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:48.077338 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:21:48.108726 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:48.108677 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" podStartSLOduration=1.108657991 podStartE2EDuration="1.108657991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:48.091715406 +0000 UTC m=+790.137040386" watchObservedRunningTime="2026-09-11 15:21:48.108657991 +0000 UTC m=+790.153982972" Sep 11 15:21:48.109994 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:48.109961 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" podStartSLOduration=1.109948424 podStartE2EDuration="1.109948424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:48.107959658 +0000 UTC m=+790.153284640" watchObservedRunningTime="2026-09-11 15:21:48.109948424 +0000 UTC m=+790.155273449" Sep 11 15:21:49.015965 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:49.015927 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:21:54.047870 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:54.047841 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:21:54.048260 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:54.047915 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:21:57.679161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.679127 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg"] Sep 11 15:21:57.682785 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.682761 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:21:57.685097 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.685071 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9f08bd\"" Sep 11 15:21:57.685461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.685444 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9f08bd-dockercfg-gc5n8\"" Sep 11 15:21:57.696093 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.696068 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg"] Sep 11 15:21:57.729690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.729658 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f"] Sep 11 15:21:57.733310 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.733286 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:21:57.743302 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.743276 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f"] Sep 11 15:21:57.823183 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.823144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc6eed3-1e2c-48af-87b9-7833e2206495-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg\" (UID: \"1bc6eed3-1e2c-48af-87b9-7833e2206495\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:21:57.823374 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.823199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjmkg\" (UniqueName: \"kubernetes.io/projected/bfb087d5-d6f0-43d3-b38f-44900016a9e9-kube-api-access-kjmkg\") pod \"payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f\" (UID: \"bfb087d5-d6f0-43d3-b38f-44900016a9e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:21:57.823374 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.823309 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb087d5-d6f0-43d3-b38f-44900016a9e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f\" (UID: \"bfb087d5-d6f0-43d3-b38f-44900016a9e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:21:57.823374 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.823349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqjs\" (UniqueName: \"kubernetes.io/projected/1bc6eed3-1e2c-48af-87b9-7833e2206495-kube-api-access-hzqjs\") pod \"payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg\" (UID: \"1bc6eed3-1e2c-48af-87b9-7833e2206495\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:21:57.923732 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.923699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc6eed3-1e2c-48af-87b9-7833e2206495-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg\" (UID: \"1bc6eed3-1e2c-48af-87b9-7833e2206495\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:21:57.923905 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.923754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjmkg\" (UniqueName: \"kubernetes.io/projected/bfb087d5-d6f0-43d3-b38f-44900016a9e9-kube-api-access-kjmkg\") pod \"payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f\" (UID: \"bfb087d5-d6f0-43d3-b38f-44900016a9e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:21:57.923905 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.923863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb087d5-d6f0-43d3-b38f-44900016a9e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f\" (UID: \"bfb087d5-d6f0-43d3-b38f-44900016a9e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:21:57.923905 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.923901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqjs\" (UniqueName: \"kubernetes.io/projected/1bc6eed3-1e2c-48af-87b9-7833e2206495-kube-api-access-hzqjs\") pod \"payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg\" (UID: \"1bc6eed3-1e2c-48af-87b9-7833e2206495\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:21:57.924588 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.924567 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb087d5-d6f0-43d3-b38f-44900016a9e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f\" (UID: \"bfb087d5-d6f0-43d3-b38f-44900016a9e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:21:57.924662 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.924606 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc6eed3-1e2c-48af-87b9-7833e2206495-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg\" (UID: \"1bc6eed3-1e2c-48af-87b9-7833e2206495\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:21:57.932402 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.932340 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjmkg\" (UniqueName: \"kubernetes.io/projected/bfb087d5-d6f0-43d3-b38f-44900016a9e9-kube-api-access-kjmkg\") pod \"payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f\" (UID: \"bfb087d5-d6f0-43d3-b38f-44900016a9e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:21:57.932402 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.932343 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqjs\" (UniqueName: \"kubernetes.io/projected/1bc6eed3-1e2c-48af-87b9-7833e2206495-kube-api-access-hzqjs\") pod \"payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg\" (UID: \"1bc6eed3-1e2c-48af-87b9-7833e2206495\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:21:57.995654 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:57.995630 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:21:58.043892 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:58.043856 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:21:58.276327 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:58.276255 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg"] Sep 11 15:21:58.279379 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:21:58.279354 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc6eed3_1e2c_48af_87b9_7833e2206495.slice/crio-46afcb989a3344a51dca80af9a81255580cced6cc1a6cfb7123bb9a31e4f905b WatchSource:0}: Error finding container 46afcb989a3344a51dca80af9a81255580cced6cc1a6cfb7123bb9a31e4f905b: Status 404 returned error can't find the container with id 46afcb989a3344a51dca80af9a81255580cced6cc1a6cfb7123bb9a31e4f905b Sep 11 15:21:58.331490 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:58.331468 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f"] Sep 11 15:21:58.333509 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:21:58.333484 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb087d5_d6f0_43d3_b38f_44900016a9e9.slice/crio-9502ca005460cb377d6a64f4bd64ed815d5f6a3a60f0d32c494fd05091566114 WatchSource:0}: Error finding container 9502ca005460cb377d6a64f4bd64ed815d5f6a3a60f0d32c494fd05091566114: Status 404 returned error can't find the container with id 9502ca005460cb377d6a64f4bd64ed815d5f6a3a60f0d32c494fd05091566114 Sep 11 15:21:59.086467 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:59.086438 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:21:59.086918 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:59.086677 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:21:59.138854 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:59.138575 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" event={"ID":"bfb087d5-d6f0-43d3-b38f-44900016a9e9","Type":"ContainerStarted","Data":"198eb0b2c3c97ca9b20c03b542ecb8bf0cb2c092ce1b689131b34fcf861da3cc"} Sep 11 15:21:59.138854 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:59.138632 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" event={"ID":"bfb087d5-d6f0-43d3-b38f-44900016a9e9","Type":"ContainerStarted","Data":"9502ca005460cb377d6a64f4bd64ed815d5f6a3a60f0d32c494fd05091566114"} Sep 11 15:21:59.138854 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:59.138749 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:21:59.140547 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:59.140517 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" event={"ID":"1bc6eed3-1e2c-48af-87b9-7833e2206495","Type":"ContainerStarted","Data":"041579c33527729dd436625636477d5a2d2351a8a2b0fc0fd9a17b97322da64c"} Sep 11 15:21:59.140672 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:59.140566 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" event={"ID":"1bc6eed3-1e2c-48af-87b9-7833e2206495","Type":"ContainerStarted","Data":"46afcb989a3344a51dca80af9a81255580cced6cc1a6cfb7123bb9a31e4f905b"} Sep 11 15:21:59.140672 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:59.140610 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:21:59.159808 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:59.159734 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" podStartSLOduration=2.159717033 podStartE2EDuration="2.159717033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:59.156365471 +0000 UTC m=+801.201690454" watchObservedRunningTime="2026-09-11 15:21:59.159717033 +0000 UTC m=+801.205042014" Sep 11 15:21:59.175897 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:21:59.175851 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" podStartSLOduration=2.17583353 podStartE2EDuration="2.17583353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:59.1754698 +0000 UTC m=+801.220794783" watchObservedRunningTime="2026-09-11 15:21:59.17583353 +0000 UTC m=+801.221158513" Sep 11 15:22:00.342602 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:00.342530 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs"] Sep 11 15:22:00.346134 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:00.346118 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" Sep 11 15:22:00.349906 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:00.349888 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:22:00.353351 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:00.353330 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:22:00.359493 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:00.359461 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs"] Sep 11 15:22:00.448007 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:00.447984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6940b1a-8fc0-40a0-84ab-92372ccf7638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs\" (UID: \"e6940b1a-8fc0-40a0-84ab-92372ccf7638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" Sep 11 15:22:00.548747 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:00.548703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6940b1a-8fc0-40a0-84ab-92372ccf7638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs\" (UID: \"e6940b1a-8fc0-40a0-84ab-92372ccf7638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" Sep 11 15:22:00.549319 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:00.549299 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6940b1a-8fc0-40a0-84ab-92372ccf7638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs\" (UID: \"e6940b1a-8fc0-40a0-84ab-92372ccf7638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" Sep 11 15:22:00.656902 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:00.656837 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" Sep 11 15:22:00.929357 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:00.929335 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs"] Sep 11 15:22:00.931020 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:22:00.930994 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6940b1a_8fc0_40a0_84ab_92372ccf7638.slice/crio-e1900fd84a64456a59d4c5a9038974d14a824743b16d1d8aba5d5fea62248e05 WatchSource:0}: Error finding container e1900fd84a64456a59d4c5a9038974d14a824743b16d1d8aba5d5fea62248e05: Status 404 returned error can't find the container with id e1900fd84a64456a59d4c5a9038974d14a824743b16d1d8aba5d5fea62248e05 Sep 11 15:22:01.151698 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:01.151658 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" event={"ID":"e6940b1a-8fc0-40a0-84ab-92372ccf7638","Type":"ContainerStarted","Data":"e1900fd84a64456a59d4c5a9038974d14a824743b16d1d8aba5d5fea62248e05"} Sep 11 15:22:04.168748 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:04.168713 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" event={"ID":"e6940b1a-8fc0-40a0-84ab-92372ccf7638","Type":"ContainerStarted","Data":"67af3366fc2d6996360d33f26c7b340597d352c6ef3b450907fd74350e3b8ccf"} Sep 11 15:22:04.197506 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:04.197459 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" podStartSLOduration=1.633144014 podStartE2EDuration="4.197445629s" totalImagesPullingTime="2.564301615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:22:04.195156482 +0000 UTC m=+806.240481466" watchObservedRunningTime="2026-09-11 15:22:04.197445629 +0000 UTC m=+806.242770610" Sep 11 15:22:05.175491 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:05.175456 2573 generic.go:356] "Generic (PLEG): container finished" podID="e6940b1a-8fc0-40a0-84ab-92372ccf7638" containerID="67af3366fc2d6996360d33f26c7b340597d352c6ef3b450907fd74350e3b8ccf" exitCode=0 Sep 11 15:22:05.175866 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:05.175545 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" event={"ID":"e6940b1a-8fc0-40a0-84ab-92372ccf7638","Type":"ContainerDied","Data":"67af3366fc2d6996360d33f26c7b340597d352c6ef3b450907fd74350e3b8ccf"} Sep 11 15:22:06.315507 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:06.315483 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" Sep 11 15:22:06.403488 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:06.403459 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6940b1a-8fc0-40a0-84ab-92372ccf7638-maas-api-service-ca\") pod \"e6940b1a-8fc0-40a0-84ab-92372ccf7638\" (UID: \"e6940b1a-8fc0-40a0-84ab-92372ccf7638\") " Sep 11 15:22:06.403840 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:06.403819 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6940b1a-8fc0-40a0-84ab-92372ccf7638-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6940b1a-8fc0-40a0-84ab-92372ccf7638" (UID: "e6940b1a-8fc0-40a0-84ab-92372ccf7638"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:22:06.504309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:06.504283 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6940b1a-8fc0-40a0-84ab-92372ccf7638-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:22:07.186523 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:07.186490 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" event={"ID":"e6940b1a-8fc0-40a0-84ab-92372ccf7638","Type":"ContainerDied","Data":"e1900fd84a64456a59d4c5a9038974d14a824743b16d1d8aba5d5fea62248e05"} Sep 11 15:22:07.186691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:07.186569 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1900fd84a64456a59d4c5a9038974d14a824743b16d1d8aba5d5fea62248e05" Sep 11 15:22:07.186691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:07.186499 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs" Sep 11 15:22:10.147636 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:10.147608 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:22:10.147985 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:10.147734 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:22:11.244684 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.244651 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd"] Sep 11 15:22:11.245149 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.244962 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" podUID="c5a7b401-9cf0-4853-b4f8-cf3766e35e8c" containerName="payload-processing" containerID="cri-o://96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737" gracePeriod=30 Sep 11 15:22:11.254607 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.254583 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd"] Sep 11 15:22:11.254845 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.254805 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" podUID="4fe688c0-17dd-47a3-bf91-ddeb00a40a90" containerName="payload-pre-processing" containerID="cri-o://fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b" gracePeriod=30 Sep 11 15:22:11.427229 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.427197 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs"] Sep 11 15:22:11.429811 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.429764 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e6b7dd7-42nvs"] Sep 11 15:22:11.517447 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.517423 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:22:11.551961 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.551933 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:22:11.647580 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.647552 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-plugins-config-volume\") pod \"4fe688c0-17dd-47a3-bf91-ddeb00a40a90\" (UID: \"4fe688c0-17dd-47a3-bf91-ddeb00a40a90\") " Sep 11 15:22:11.647704 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.647666 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg7f8\" (UniqueName: \"kubernetes.io/projected/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-kube-api-access-vg7f8\") pod \"4fe688c0-17dd-47a3-bf91-ddeb00a40a90\" (UID: \"4fe688c0-17dd-47a3-bf91-ddeb00a40a90\") " Sep 11 15:22:11.647933 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.647910 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fe688c0-17dd-47a3-bf91-ddeb00a40a90" (UID: "4fe688c0-17dd-47a3-bf91-ddeb00a40a90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:22:11.648015 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.648000 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:22:11.650043 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.650024 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-kube-api-access-vg7f8" (OuterVolumeSpecName: "kube-api-access-vg7f8") pod "4fe688c0-17dd-47a3-bf91-ddeb00a40a90" (UID: "4fe688c0-17dd-47a3-bf91-ddeb00a40a90"). InnerVolumeSpecName "kube-api-access-vg7f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:22:11.748717 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.748696 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-plugins-config-volume\") pod \"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c\" (UID: \"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c\") " Sep 11 15:22:11.748849 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.748838 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlqtv\" (UniqueName: \"kubernetes.io/projected/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-kube-api-access-jlqtv\") pod \"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c\" (UID: \"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c\") " Sep 11 15:22:11.749044 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.748990 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg7f8\" (UniqueName: \"kubernetes.io/projected/4fe688c0-17dd-47a3-bf91-ddeb00a40a90-kube-api-access-vg7f8\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:22:11.749125 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.749087 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5a7b401-9cf0-4853-b4f8-cf3766e35e8c" (UID: "c5a7b401-9cf0-4853-b4f8-cf3766e35e8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:22:11.750743 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.750721 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-kube-api-access-jlqtv" (OuterVolumeSpecName: "kube-api-access-jlqtv") pod "c5a7b401-9cf0-4853-b4f8-cf3766e35e8c" (UID: "c5a7b401-9cf0-4853-b4f8-cf3766e35e8c"). InnerVolumeSpecName "kube-api-access-jlqtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:22:11.850062 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.850030 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlqtv\" (UniqueName: \"kubernetes.io/projected/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-kube-api-access-jlqtv\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:22:11.850062 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:11.850061 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:22:12.215357 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.215315 2573 generic.go:356] "Generic (PLEG): container finished" podID="c5a7b401-9cf0-4853-b4f8-cf3766e35e8c" containerID="96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737" exitCode=0 Sep 11 15:22:12.215530 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.215374 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" Sep 11 15:22:12.215530 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.215398 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" event={"ID":"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c","Type":"ContainerDied","Data":"96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737"} Sep 11 15:22:12.215530 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.215479 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd" event={"ID":"c5a7b401-9cf0-4853-b4f8-cf3766e35e8c","Type":"ContainerDied","Data":"bf9aa4a72e65d1f255f5fbde2fb79a3a0477581f49cd676131b5b73b4d39a5de"} Sep 11 15:22:12.215530 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.215503 2573 scope.go:117] "RemoveContainer" containerID="96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737" Sep 11 15:22:12.216860 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.216838 2573 generic.go:356] "Generic (PLEG): container finished" podID="4fe688c0-17dd-47a3-bf91-ddeb00a40a90" containerID="fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b" exitCode=0 Sep 11 15:22:12.216982 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.216916 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" Sep 11 15:22:12.216982 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.216935 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" event={"ID":"4fe688c0-17dd-47a3-bf91-ddeb00a40a90","Type":"ContainerDied","Data":"fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b"} Sep 11 15:22:12.217090 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.216988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd" event={"ID":"4fe688c0-17dd-47a3-bf91-ddeb00a40a90","Type":"ContainerDied","Data":"ab7653d564e8fde35e916ca0226a3873524df6eb1dee1501110d52b5ec8e9019"} Sep 11 15:22:12.224209 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.224193 2573 scope.go:117] "RemoveContainer" containerID="96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737" Sep 11 15:22:12.224469 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:22:12.224453 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737\": container with ID starting with 96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737 not found: ID does not exist" containerID="96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737" Sep 11 15:22:12.224535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.224479 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737"} err="failed to get container status \"96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737\": rpc error: code = NotFound desc = could not find container \"96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737\": container with ID starting with 96b0fd6ab5e0f4840839c5e3cd5c2482b409c8011295d5ba44121f5af25e3737 not found: ID does not exist" Sep 11 15:22:12.224535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.224501 2573 scope.go:117] "RemoveContainer" containerID="fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b" Sep 11 15:22:12.232568 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.232545 2573 scope.go:117] "RemoveContainer" containerID="fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b" Sep 11 15:22:12.232830 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:22:12.232811 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b\": container with ID starting with fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b not found: ID does not exist" containerID="fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b" Sep 11 15:22:12.232925 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.232834 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b"} err="failed to get container status \"fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b\": rpc error: code = NotFound desc = could not find container \"fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b\": container with ID starting with fadc7483b37097ba3affb9ee65ad659c7955d21f576ffae388a3942ec4abab0b not found: ID does not exist" Sep 11 15:22:12.245766 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.245735 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd"] Sep 11 15:22:12.250654 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.250632 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e6b7dd7-57b4dd7bfd-26hjd"] Sep 11 15:22:12.261344 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.261323 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd"] Sep 11 15:22:12.265432 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.265254 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e6b7dd7-f4d466ccc-cxnqd"] Sep 11 15:22:12.494317 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.494247 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe688c0-17dd-47a3-bf91-ddeb00a40a90" path="/var/lib/kubelet/pods/4fe688c0-17dd-47a3-bf91-ddeb00a40a90/volumes" Sep 11 15:22:12.494601 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.494588 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a7b401-9cf0-4853-b4f8-cf3766e35e8c" path="/var/lib/kubelet/pods/c5a7b401-9cf0-4853-b4f8-cf3766e35e8c/volumes" Sep 11 15:22:12.494906 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:12.494893 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6940b1a-8fc0-40a0-84ab-92372ccf7638" path="/var/lib/kubelet/pods/e6940b1a-8fc0-40a0-84ab-92372ccf7638/volumes" Sep 11 15:22:18.711600 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.711568 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl"] Sep 11 15:22:18.712046 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.711994 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a7b401-9cf0-4853-b4f8-cf3766e35e8c" containerName="payload-processing" Sep 11 15:22:18.712046 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.712007 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a7b401-9cf0-4853-b4f8-cf3766e35e8c" containerName="payload-processing" Sep 11 15:22:18.712046 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.712017 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6940b1a-8fc0-40a0-84ab-92372ccf7638" containerName="revoke-keys" Sep 11 15:22:18.712046 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.712022 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6940b1a-8fc0-40a0-84ab-92372ccf7638" containerName="revoke-keys" Sep 11 15:22:18.712046 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.712038 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe688c0-17dd-47a3-bf91-ddeb00a40a90" containerName="payload-pre-processing" Sep 11 15:22:18.712046 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.712043 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe688c0-17dd-47a3-bf91-ddeb00a40a90" containerName="payload-pre-processing" Sep 11 15:22:18.712237 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.712102 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe688c0-17dd-47a3-bf91-ddeb00a40a90" containerName="payload-pre-processing" Sep 11 15:22:18.712237 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.712114 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a7b401-9cf0-4853-b4f8-cf3766e35e8c" containerName="payload-processing" Sep 11 15:22:18.712237 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.712121 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6940b1a-8fc0-40a0-84ab-92372ccf7638" containerName="revoke-keys" Sep 11 15:22:18.716686 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.716663 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:18.718946 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.718915 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-73b24789-dockercfg-d6brd\"" Sep 11 15:22:18.719054 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.719036 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-73b24789\"" Sep 11 15:22:18.725952 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.725927 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl"] Sep 11 15:22:18.805635 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.805610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl\" (UID: \"016bca0e-2aa0-47b4-a1a7-acc1a26322f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:18.805763 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.805724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l7rk\" (UniqueName: \"kubernetes.io/projected/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-kube-api-access-5l7rk\") pod \"payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl\" (UID: \"016bca0e-2aa0-47b4-a1a7-acc1a26322f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:18.906694 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.906663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl\" (UID: \"016bca0e-2aa0-47b4-a1a7-acc1a26322f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:18.906824 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.906743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l7rk\" (UniqueName: \"kubernetes.io/projected/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-kube-api-access-5l7rk\") pod \"payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl\" (UID: \"016bca0e-2aa0-47b4-a1a7-acc1a26322f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:18.907349 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.907330 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl\" (UID: \"016bca0e-2aa0-47b4-a1a7-acc1a26322f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:18.919173 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:18.919150 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l7rk\" (UniqueName: \"kubernetes.io/projected/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-kube-api-access-5l7rk\") pod \"payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl\" (UID: \"016bca0e-2aa0-47b4-a1a7-acc1a26322f8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:19.028614 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:19.028558 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:19.300343 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:19.300317 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl"] Sep 11 15:22:19.302919 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:22:19.302891 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016bca0e_2aa0_47b4_a1a7_acc1a26322f8.slice/crio-29911bf1db8365f4b2c186d508404e136ae47ff597337d8bfa4a833a80980be5 WatchSource:0}: Error finding container 29911bf1db8365f4b2c186d508404e136ae47ff597337d8bfa4a833a80980be5: Status 404 returned error can't find the container with id 29911bf1db8365f4b2c186d508404e136ae47ff597337d8bfa4a833a80980be5 Sep 11 15:22:20.262143 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:20.262109 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" event={"ID":"016bca0e-2aa0-47b4-a1a7-acc1a26322f8","Type":"ContainerStarted","Data":"392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9"} Sep 11 15:22:20.262496 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:20.262241 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:20.262496 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:20.262263 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" event={"ID":"016bca0e-2aa0-47b4-a1a7-acc1a26322f8","Type":"ContainerStarted","Data":"29911bf1db8365f4b2c186d508404e136ae47ff597337d8bfa4a833a80980be5"} Sep 11 15:22:20.281873 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:20.281758 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" podStartSLOduration=2.281743624 podStartE2EDuration="2.281743624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:22:20.281387713 +0000 UTC m=+822.326712695" watchObservedRunningTime="2026-09-11 15:22:20.281743624 +0000 UTC m=+822.327068637" Sep 11 15:22:23.373574 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.373539 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2"] Sep 11 15:22:23.377235 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.377219 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.379564 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.379543 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-91304794-kserve-self-signed-certs\"" Sep 11 15:22:23.387214 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.387191 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2"] Sep 11 15:22:23.443540 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.443519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-dshm\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.443640 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.443563 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tmp-dir\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.443640 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.443582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-home\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.443714 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.443661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tls-certs\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.443714 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.443694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzhjr\" (UniqueName: \"kubernetes.io/projected/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-kube-api-access-kzhjr\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.443806 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.443716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-model-cache\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.544812 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.544768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-dshm\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.544945 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.544835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tmp-dir\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.544945 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.544853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-home\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.544945 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.544885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tls-certs\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.544945 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.544908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzhjr\" (UniqueName: \"kubernetes.io/projected/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-kube-api-access-kzhjr\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.544945 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.544933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-model-cache\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.545228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.545192 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-model-cache\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.545598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.545571 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tmp-dir\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.545710 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.545669 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-home\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.547848 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.547825 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-dshm\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.547994 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.547976 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tls-certs\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.553637 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.553613 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzhjr\" (UniqueName: \"kubernetes.io/projected/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-kube-api-access-kzhjr\") pod \"e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.696798 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.696722 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:23.969694 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:23.969672 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2"] Sep 11 15:22:23.971514 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:22:23.971488 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c8ed830_dffe_4e2a_ab54_3ef3201305a7.slice/crio-a852e4479d8a88785193c0c571186561ccff1cb9674e4e303847f8f1891b6018 WatchSource:0}: Error finding container a852e4479d8a88785193c0c571186561ccff1cb9674e4e303847f8f1891b6018: Status 404 returned error can't find the container with id a852e4479d8a88785193c0c571186561ccff1cb9674e4e303847f8f1891b6018 Sep 11 15:22:24.284020 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:24.283935 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" event={"ID":"9c8ed830-dffe-4e2a-ab54-3ef3201305a7","Type":"ContainerStarted","Data":"c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73"} Sep 11 15:22:24.284020 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:24.283988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" event={"ID":"9c8ed830-dffe-4e2a-ab54-3ef3201305a7","Type":"ContainerStarted","Data":"a852e4479d8a88785193c0c571186561ccff1cb9674e4e303847f8f1891b6018"} Sep 11 15:22:24.304706 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:24.304654 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" podStartSLOduration=1.304640058 podStartE2EDuration="1.304640058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:22:24.302711076 +0000 UTC m=+826.348036048" watchObservedRunningTime="2026-09-11 15:22:24.304640058 +0000 UTC m=+826.349965038" Sep 11 15:22:31.270119 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:31.270080 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:33.697259 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:33.697222 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:33.697620 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:33.697275 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:33.709739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:33.709714 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:34.344654 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:34.344620 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:22:40.802343 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:40.802310 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s"] Sep 11 15:22:40.811116 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:40.811092 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s" Sep 11 15:22:40.812536 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:40.812511 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s"] Sep 11 15:22:40.813395 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:40.813374 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:22:40.813560 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:40.813387 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:22:40.890860 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:40.890835 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/317d90d8-45ff-4582-9c9b-b5438cbc078b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s\" (UID: \"317d90d8-45ff-4582-9c9b-b5438cbc078b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s" Sep 11 15:22:40.992168 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:40.992132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/317d90d8-45ff-4582-9c9b-b5438cbc078b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s\" (UID: \"317d90d8-45ff-4582-9c9b-b5438cbc078b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s" Sep 11 15:22:40.992681 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:40.992663 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/317d90d8-45ff-4582-9c9b-b5438cbc078b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s\" (UID: \"317d90d8-45ff-4582-9c9b-b5438cbc078b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s" Sep 11 15:22:41.123292 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:41.123268 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s" Sep 11 15:22:41.392335 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:41.392313 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s"] Sep 11 15:22:41.394134 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:22:41.394090 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317d90d8_45ff_4582_9c9b_b5438cbc078b.slice/crio-836e94ff9d0026904c081fe9226a351f477ae8bfe2671638600c167699764069 WatchSource:0}: Error finding container 836e94ff9d0026904c081fe9226a351f477ae8bfe2671638600c167699764069: Status 404 returned error can't find the container with id 836e94ff9d0026904c081fe9226a351f477ae8bfe2671638600c167699764069 Sep 11 15:22:42.374210 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:42.374174 2573 generic.go:356] "Generic (PLEG): container finished" podID="317d90d8-45ff-4582-9c9b-b5438cbc078b" containerID="d7819f9d2a5a3e021942a5c63d26055494b51893ccbec851913cb4eaa6273f15" exitCode=0 Sep 11 15:22:42.374600 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:42.374245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s" event={"ID":"317d90d8-45ff-4582-9c9b-b5438cbc078b","Type":"ContainerDied","Data":"d7819f9d2a5a3e021942a5c63d26055494b51893ccbec851913cb4eaa6273f15"} Sep 11 15:22:42.374600 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:42.374285 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s" event={"ID":"317d90d8-45ff-4582-9c9b-b5438cbc078b","Type":"ContainerStarted","Data":"836e94ff9d0026904c081fe9226a351f477ae8bfe2671638600c167699764069"} Sep 11 15:22:43.512991 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:43.512964 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s" Sep 11 15:22:43.613395 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:43.613369 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/317d90d8-45ff-4582-9c9b-b5438cbc078b-maas-api-service-ca\") pod \"317d90d8-45ff-4582-9c9b-b5438cbc078b\" (UID: \"317d90d8-45ff-4582-9c9b-b5438cbc078b\") " Sep 11 15:22:43.613753 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:43.613726 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317d90d8-45ff-4582-9c9b-b5438cbc078b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "317d90d8-45ff-4582-9c9b-b5438cbc078b" (UID: "317d90d8-45ff-4582-9c9b-b5438cbc078b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:22:43.714276 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:43.714217 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/317d90d8-45ff-4582-9c9b-b5438cbc078b-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:22:44.385858 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:44.385822 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s" event={"ID":"317d90d8-45ff-4582-9c9b-b5438cbc078b","Type":"ContainerDied","Data":"836e94ff9d0026904c081fe9226a351f477ae8bfe2671638600c167699764069"} Sep 11 15:22:44.386068 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:44.385892 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="836e94ff9d0026904c081fe9226a351f477ae8bfe2671638600c167699764069" Sep 11 15:22:44.386068 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:44.385830 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s" Sep 11 15:22:47.875100 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:47.875070 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl"] Sep 11 15:22:47.875637 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:47.875606 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" podUID="016bca0e-2aa0-47b4-a1a7-acc1a26322f8" containerName="payload-pre-processing" containerID="cri-o://392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9" gracePeriod=30 Sep 11 15:22:48.057234 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.057203 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s"] Sep 11 15:22:48.069093 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.068261 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73b24789-x6x9s"] Sep 11 15:22:48.132690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.132641 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:48.247897 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.247863 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-plugins-config-volume\") pod \"016bca0e-2aa0-47b4-a1a7-acc1a26322f8\" (UID: \"016bca0e-2aa0-47b4-a1a7-acc1a26322f8\") " Sep 11 15:22:48.247897 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.247897 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l7rk\" (UniqueName: \"kubernetes.io/projected/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-kube-api-access-5l7rk\") pod \"016bca0e-2aa0-47b4-a1a7-acc1a26322f8\" (UID: \"016bca0e-2aa0-47b4-a1a7-acc1a26322f8\") " Sep 11 15:22:48.248218 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.248193 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "016bca0e-2aa0-47b4-a1a7-acc1a26322f8" (UID: "016bca0e-2aa0-47b4-a1a7-acc1a26322f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:22:48.250103 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.250071 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-kube-api-access-5l7rk" (OuterVolumeSpecName: "kube-api-access-5l7rk") pod "016bca0e-2aa0-47b4-a1a7-acc1a26322f8" (UID: "016bca0e-2aa0-47b4-a1a7-acc1a26322f8"). InnerVolumeSpecName "kube-api-access-5l7rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:22:48.349374 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.349349 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:22:48.349374 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.349372 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l7rk\" (UniqueName: \"kubernetes.io/projected/016bca0e-2aa0-47b4-a1a7-acc1a26322f8-kube-api-access-5l7rk\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:22:48.407529 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.407474 2573 generic.go:356] "Generic (PLEG): container finished" podID="016bca0e-2aa0-47b4-a1a7-acc1a26322f8" containerID="392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9" exitCode=0 Sep 11 15:22:48.407529 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.407518 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" Sep 11 15:22:48.407669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.407542 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" event={"ID":"016bca0e-2aa0-47b4-a1a7-acc1a26322f8","Type":"ContainerDied","Data":"392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9"} Sep 11 15:22:48.407669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.407605 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl" event={"ID":"016bca0e-2aa0-47b4-a1a7-acc1a26322f8","Type":"ContainerDied","Data":"29911bf1db8365f4b2c186d508404e136ae47ff597337d8bfa4a833a80980be5"} Sep 11 15:22:48.407669 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.407626 2573 scope.go:117] "RemoveContainer" containerID="392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9" Sep 11 15:22:48.416562 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.416546 2573 scope.go:117] "RemoveContainer" containerID="392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9" Sep 11 15:22:48.416898 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:22:48.416869 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9\": container with ID starting with 392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9 not found: ID does not exist" containerID="392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9" Sep 11 15:22:48.416975 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.416911 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9"} err="failed to get container status \"392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9\": rpc error: code = NotFound desc = could not find container \"392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9\": container with ID starting with 392b17f6424c8ba016cfee62c3b0faa63ee563a4d76625e64fcf58b5483f2dd9 not found: ID does not exist" Sep 11 15:22:48.431400 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.431379 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl"] Sep 11 15:22:48.442861 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.442840 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-73b24789-576975fb98-vfpjl"] Sep 11 15:22:48.493202 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.493179 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="016bca0e-2aa0-47b4-a1a7-acc1a26322f8" path="/var/lib/kubelet/pods/016bca0e-2aa0-47b4-a1a7-acc1a26322f8/volumes" Sep 11 15:22:48.493554 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:48.493541 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="317d90d8-45ff-4582-9c9b-b5438cbc078b" path="/var/lib/kubelet/pods/317d90d8-45ff-4582-9c9b-b5438cbc078b/volumes" Sep 11 15:22:56.223991 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.223959 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2"] Sep 11 15:22:56.224646 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.224604 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="016bca0e-2aa0-47b4-a1a7-acc1a26322f8" containerName="payload-pre-processing" Sep 11 15:22:56.224646 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.224634 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="016bca0e-2aa0-47b4-a1a7-acc1a26322f8" containerName="payload-pre-processing" Sep 11 15:22:56.224833 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.224652 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="317d90d8-45ff-4582-9c9b-b5438cbc078b" containerName="revoke-keys" Sep 11 15:22:56.224833 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.224661 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="317d90d8-45ff-4582-9c9b-b5438cbc078b" containerName="revoke-keys" Sep 11 15:22:56.224833 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.224790 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="317d90d8-45ff-4582-9c9b-b5438cbc078b" containerName="revoke-keys" Sep 11 15:22:56.224833 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.224803 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="016bca0e-2aa0-47b4-a1a7-acc1a26322f8" containerName="payload-pre-processing" Sep 11 15:22:56.229331 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.229315 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:22:56.231455 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.231438 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8c097d73\"" Sep 11 15:22:56.238288 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.238261 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2"] Sep 11 15:22:56.318430 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.318395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb4ct\" (UniqueName: \"kubernetes.io/projected/cd166718-2b5a-4b41-b9ff-d63faf789c0b-kube-api-access-bb4ct\") pod \"payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2\" (UID: \"cd166718-2b5a-4b41-b9ff-d63faf789c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:22:56.318549 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.318434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd166718-2b5a-4b41-b9ff-d63faf789c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2\" (UID: \"cd166718-2b5a-4b41-b9ff-d63faf789c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:22:56.419094 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.419069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb4ct\" (UniqueName: \"kubernetes.io/projected/cd166718-2b5a-4b41-b9ff-d63faf789c0b-kube-api-access-bb4ct\") pod \"payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2\" (UID: \"cd166718-2b5a-4b41-b9ff-d63faf789c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:22:56.419229 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.419109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd166718-2b5a-4b41-b9ff-d63faf789c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2\" (UID: \"cd166718-2b5a-4b41-b9ff-d63faf789c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:22:56.419805 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.419751 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd166718-2b5a-4b41-b9ff-d63faf789c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2\" (UID: \"cd166718-2b5a-4b41-b9ff-d63faf789c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:22:56.427387 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.427364 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb4ct\" (UniqueName: \"kubernetes.io/projected/cd166718-2b5a-4b41-b9ff-d63faf789c0b-kube-api-access-bb4ct\") pod \"payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2\" (UID: \"cd166718-2b5a-4b41-b9ff-d63faf789c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:22:56.540822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.540754 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:22:56.812042 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:56.812013 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2"] Sep 11 15:22:56.813549 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:22:56.813518 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd166718_2b5a_4b41_b9ff_d63faf789c0b.slice/crio-41bf8767da822bdab7b0d51a2df3b171efe0afcbb9203b1cb0fa76b03e919b7d WatchSource:0}: Error finding container 41bf8767da822bdab7b0d51a2df3b171efe0afcbb9203b1cb0fa76b03e919b7d: Status 404 returned error can't find the container with id 41bf8767da822bdab7b0d51a2df3b171efe0afcbb9203b1cb0fa76b03e919b7d Sep 11 15:22:57.456185 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:57.456152 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" event={"ID":"cd166718-2b5a-4b41-b9ff-d63faf789c0b","Type":"ContainerStarted","Data":"73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d"} Sep 11 15:22:57.456532 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:57.456201 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" event={"ID":"cd166718-2b5a-4b41-b9ff-d63faf789c0b","Type":"ContainerStarted","Data":"41bf8767da822bdab7b0d51a2df3b171efe0afcbb9203b1cb0fa76b03e919b7d"} Sep 11 15:22:57.456532 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:57.456253 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:22:57.474792 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:22:57.474725 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" podStartSLOduration=1.474713876 podStartE2EDuration="1.474713876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:22:57.473089952 +0000 UTC m=+859.518414932" watchObservedRunningTime="2026-09-11 15:22:57.474713876 +0000 UTC m=+859.520038856" Sep 11 15:23:08.464110 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:08.464077 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:23:22.071362 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.071331 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92"] Sep 11 15:23:22.075195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.075177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92" Sep 11 15:23:22.078221 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.078200 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:23:22.078320 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.078199 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:23:22.083082 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.082765 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92"] Sep 11 15:23:22.239693 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.239654 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53427c8-c6ef-479a-8da4-b9d3b022439b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-34c71e-q9z92\" (UID: \"a53427c8-c6ef-479a-8da4-b9d3b022439b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92" Sep 11 15:23:22.340603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.340567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53427c8-c6ef-479a-8da4-b9d3b022439b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-34c71e-q9z92\" (UID: \"a53427c8-c6ef-479a-8da4-b9d3b022439b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92" Sep 11 15:23:22.341214 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.341189 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53427c8-c6ef-479a-8da4-b9d3b022439b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-34c71e-q9z92\" (UID: \"a53427c8-c6ef-479a-8da4-b9d3b022439b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92" Sep 11 15:23:22.386613 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.386584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92" Sep 11 15:23:22.657291 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.657263 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92"] Sep 11 15:23:22.659333 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:23:22.659304 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53427c8_c6ef_479a_8da4_b9d3b022439b.slice/crio-070f3453e9068ef648d6265c65875b384359c0d2652bd86ffe3d8cb946cb2758 WatchSource:0}: Error finding container 070f3453e9068ef648d6265c65875b384359c0d2652bd86ffe3d8cb946cb2758: Status 404 returned error can't find the container with id 070f3453e9068ef648d6265c65875b384359c0d2652bd86ffe3d8cb946cb2758 Sep 11 15:23:22.807623 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.807593 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r"] Sep 11 15:23:22.812390 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.812375 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r" Sep 11 15:23:22.817943 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.817922 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r"] Sep 11 15:23:22.947183 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:22.947121 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6407a83e-51ba-41e9-9072-6db665379603-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c097d73-6v82r\" (UID: \"6407a83e-51ba-41e9-9072-6db665379603\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r" Sep 11 15:23:23.048020 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:23.047992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6407a83e-51ba-41e9-9072-6db665379603-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c097d73-6v82r\" (UID: \"6407a83e-51ba-41e9-9072-6db665379603\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r" Sep 11 15:23:23.048550 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:23.048532 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6407a83e-51ba-41e9-9072-6db665379603-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c097d73-6v82r\" (UID: \"6407a83e-51ba-41e9-9072-6db665379603\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r" Sep 11 15:23:23.124290 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:23.124256 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r" Sep 11 15:23:23.399380 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:23.399355 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r"] Sep 11 15:23:23.439623 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:23:23.439595 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6407a83e_51ba_41e9_9072_6db665379603.slice/crio-92cd71fadd3541a74929eb6f9a9a0f76aa8f42830402ce28bd013d3c746ae2b1 WatchSource:0}: Error finding container 92cd71fadd3541a74929eb6f9a9a0f76aa8f42830402ce28bd013d3c746ae2b1: Status 404 returned error can't find the container with id 92cd71fadd3541a74929eb6f9a9a0f76aa8f42830402ce28bd013d3c746ae2b1 Sep 11 15:23:23.595819 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:23.595767 2573 generic.go:356] "Generic (PLEG): container finished" podID="a53427c8-c6ef-479a-8da4-b9d3b022439b" containerID="6ea39e8d21455a967be079521e0c150cc656ca3bb28e6e0ab06e3bd3aa480a3f" exitCode=0 Sep 11 15:23:23.595819 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:23.595809 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92" event={"ID":"a53427c8-c6ef-479a-8da4-b9d3b022439b","Type":"ContainerDied","Data":"6ea39e8d21455a967be079521e0c150cc656ca3bb28e6e0ab06e3bd3aa480a3f"} Sep 11 15:23:23.595965 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:23.595858 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92" event={"ID":"a53427c8-c6ef-479a-8da4-b9d3b022439b","Type":"ContainerStarted","Data":"070f3453e9068ef648d6265c65875b384359c0d2652bd86ffe3d8cb946cb2758"} Sep 11 15:23:23.597224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:23.597206 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r" event={"ID":"6407a83e-51ba-41e9-9072-6db665379603","Type":"ContainerStarted","Data":"92cd71fadd3541a74929eb6f9a9a0f76aa8f42830402ce28bd013d3c746ae2b1"} Sep 11 15:23:24.604819 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:24.604761 2573 generic.go:356] "Generic (PLEG): container finished" podID="6407a83e-51ba-41e9-9072-6db665379603" containerID="065f27410ab6c6282ebd4f148ac1bd6c3da8e141e3fe8246f8fcdb30ad148aa9" exitCode=0 Sep 11 15:23:24.604819 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:24.604802 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r" event={"ID":"6407a83e-51ba-41e9-9072-6db665379603","Type":"ContainerDied","Data":"065f27410ab6c6282ebd4f148ac1bd6c3da8e141e3fe8246f8fcdb30ad148aa9"} Sep 11 15:23:24.736310 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:24.736284 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92" Sep 11 15:23:24.763292 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:24.763268 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53427c8-c6ef-479a-8da4-b9d3b022439b-maas-api-service-ca\") pod \"a53427c8-c6ef-479a-8da4-b9d3b022439b\" (UID: \"a53427c8-c6ef-479a-8da4-b9d3b022439b\") " Sep 11 15:23:24.763591 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:24.763565 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53427c8-c6ef-479a-8da4-b9d3b022439b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a53427c8-c6ef-479a-8da4-b9d3b022439b" (UID: "a53427c8-c6ef-479a-8da4-b9d3b022439b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:23:24.864916 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:24.864863 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53427c8-c6ef-479a-8da4-b9d3b022439b-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:23:25.612338 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:25.612308 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92" Sep 11 15:23:25.612808 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:25.612353 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92" event={"ID":"a53427c8-c6ef-479a-8da4-b9d3b022439b","Type":"ContainerDied","Data":"070f3453e9068ef648d6265c65875b384359c0d2652bd86ffe3d8cb946cb2758"} Sep 11 15:23:25.612808 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:25.612394 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="070f3453e9068ef648d6265c65875b384359c0d2652bd86ffe3d8cb946cb2758" Sep 11 15:23:25.754171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:25.754143 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r" Sep 11 15:23:25.773938 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:25.773908 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6407a83e-51ba-41e9-9072-6db665379603-maas-api-service-ca\") pod \"6407a83e-51ba-41e9-9072-6db665379603\" (UID: \"6407a83e-51ba-41e9-9072-6db665379603\") " Sep 11 15:23:25.774232 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:25.774209 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6407a83e-51ba-41e9-9072-6db665379603-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6407a83e-51ba-41e9-9072-6db665379603" (UID: "6407a83e-51ba-41e9-9072-6db665379603"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:23:25.875461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:25.875395 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6407a83e-51ba-41e9-9072-6db665379603-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:23:26.618610 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:26.618585 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r" Sep 11 15:23:26.618610 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:26.618604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r" event={"ID":"6407a83e-51ba-41e9-9072-6db665379603","Type":"ContainerDied","Data":"92cd71fadd3541a74929eb6f9a9a0f76aa8f42830402ce28bd013d3c746ae2b1"} Sep 11 15:23:26.618993 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:26.618644 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92cd71fadd3541a74929eb6f9a9a0f76aa8f42830402ce28bd013d3c746ae2b1" Sep 11 15:23:32.929017 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:32.928935 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg"] Sep 11 15:23:32.929540 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:32.929176 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" podUID="9ab651a9-7564-4d22-9d39-462c40b179d2" containerName="payload-processing" containerID="cri-o://56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908" gracePeriod=30 Sep 11 15:23:32.937949 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:32.937916 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q"] Sep 11 15:23:32.938279 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:32.938221 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" podUID="63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4" containerName="payload-pre-processing" containerID="cri-o://bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5" gracePeriod=30 Sep 11 15:23:33.090732 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.090694 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92"] Sep 11 15:23:33.092585 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.092556 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-34c71e-q9z92"] Sep 11 15:23:33.208120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.208099 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:23:33.211011 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.210989 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:23:33.233253 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.233227 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjjxb\" (UniqueName: \"kubernetes.io/projected/9ab651a9-7564-4d22-9d39-462c40b179d2-kube-api-access-bjjxb\") pod \"9ab651a9-7564-4d22-9d39-462c40b179d2\" (UID: \"9ab651a9-7564-4d22-9d39-462c40b179d2\") " Sep 11 15:23:33.233396 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.233358 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmdhl\" (UniqueName: \"kubernetes.io/projected/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-kube-api-access-nmdhl\") pod \"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4\" (UID: \"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4\") " Sep 11 15:23:33.233464 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.233451 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-plugins-config-volume\") pod \"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4\" (UID: \"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4\") " Sep 11 15:23:33.233538 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.233512 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab651a9-7564-4d22-9d39-462c40b179d2-plugins-config-volume\") pod \"9ab651a9-7564-4d22-9d39-462c40b179d2\" (UID: \"9ab651a9-7564-4d22-9d39-462c40b179d2\") " Sep 11 15:23:33.234005 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.233978 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4" (UID: "63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:23:33.234325 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.234292 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab651a9-7564-4d22-9d39-462c40b179d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ab651a9-7564-4d22-9d39-462c40b179d2" (UID: "9ab651a9-7564-4d22-9d39-462c40b179d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:23:33.236101 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.236074 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab651a9-7564-4d22-9d39-462c40b179d2-kube-api-access-bjjxb" (OuterVolumeSpecName: "kube-api-access-bjjxb") pod "9ab651a9-7564-4d22-9d39-462c40b179d2" (UID: "9ab651a9-7564-4d22-9d39-462c40b179d2"). InnerVolumeSpecName "kube-api-access-bjjxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:23:33.236956 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.236930 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-kube-api-access-nmdhl" (OuterVolumeSpecName: "kube-api-access-nmdhl") pod "63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4" (UID: "63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4"). InnerVolumeSpecName "kube-api-access-nmdhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:23:33.335130 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.335103 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab651a9-7564-4d22-9d39-462c40b179d2-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:23:33.335130 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.335127 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjjxb\" (UniqueName: \"kubernetes.io/projected/9ab651a9-7564-4d22-9d39-462c40b179d2-kube-api-access-bjjxb\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:23:33.335283 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.335138 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmdhl\" (UniqueName: \"kubernetes.io/projected/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-kube-api-access-nmdhl\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:23:33.335283 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.335146 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:23:33.656912 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.656881 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ab651a9-7564-4d22-9d39-462c40b179d2" containerID="56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908" exitCode=0 Sep 11 15:23:33.657082 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.656934 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" Sep 11 15:23:33.657082 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.656972 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" event={"ID":"9ab651a9-7564-4d22-9d39-462c40b179d2","Type":"ContainerDied","Data":"56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908"} Sep 11 15:23:33.657082 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.657035 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg" event={"ID":"9ab651a9-7564-4d22-9d39-462c40b179d2","Type":"ContainerDied","Data":"fa403b4d191651f0ccb6d35623e902f616b89475dacc48f73274c2c6a7fa220b"} Sep 11 15:23:33.657082 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.657063 2573 scope.go:117] "RemoveContainer" containerID="56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908" Sep 11 15:23:33.658546 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.658517 2573 generic.go:356] "Generic (PLEG): container finished" podID="63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4" containerID="bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5" exitCode=0 Sep 11 15:23:33.658648 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.658566 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" Sep 11 15:23:33.658648 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.658572 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" event={"ID":"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4","Type":"ContainerDied","Data":"bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5"} Sep 11 15:23:33.658648 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.658622 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q" event={"ID":"63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4","Type":"ContainerDied","Data":"a444403e0dc64052cc81b82507827a24a02c1215d702869d2a7dc08037a54fbb"} Sep 11 15:23:33.667576 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.667562 2573 scope.go:117] "RemoveContainer" containerID="56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908" Sep 11 15:23:33.667905 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:23:33.667885 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908\": container with ID starting with 56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908 not found: ID does not exist" containerID="56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908" Sep 11 15:23:33.667962 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.667913 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908"} err="failed to get container status \"56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908\": rpc error: code = NotFound desc = could not find container \"56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908\": container with ID starting with 56090839feb711feb49e7d21b89226e431f8bae38cb2ea499e4e8d395436e908 not found: ID does not exist" Sep 11 15:23:33.667962 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.667945 2573 scope.go:117] "RemoveContainer" containerID="bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5" Sep 11 15:23:33.678241 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.678218 2573 scope.go:117] "RemoveContainer" containerID="bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5" Sep 11 15:23:33.678636 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:23:33.678616 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5\": container with ID starting with bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5 not found: ID does not exist" containerID="bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5" Sep 11 15:23:33.678690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.678643 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5"} err="failed to get container status \"bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5\": rpc error: code = NotFound desc = could not find container \"bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5\": container with ID starting with bbc4e059abd828c66392f2326052f4bc148301ce524c79590fcdc53135a50cd5 not found: ID does not exist" Sep 11 15:23:33.694727 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.694706 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg"] Sep 11 15:23:33.700415 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.700395 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-34c71e-6f96cddfd7-s8kcg"] Sep 11 15:23:33.714736 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.714708 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q"] Sep 11 15:23:33.719845 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:33.719826 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-34c71e-5bcf57fb46-trc4q"] Sep 11 15:23:34.493309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:34.493277 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4" path="/var/lib/kubelet/pods/63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4/volumes" Sep 11 15:23:34.493668 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:34.493616 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ab651a9-7564-4d22-9d39-462c40b179d2" path="/var/lib/kubelet/pods/9ab651a9-7564-4d22-9d39-462c40b179d2/volumes" Sep 11 15:23:34.493927 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:34.493913 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53427c8-c6ef-479a-8da4-b9d3b022439b" path="/var/lib/kubelet/pods/a53427c8-c6ef-479a-8da4-b9d3b022439b/volumes" Sep 11 15:23:35.842976 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:35.842945 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2"] Sep 11 15:23:35.843448 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:35.843184 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" podUID="cd166718-2b5a-4b41-b9ff-d63faf789c0b" containerName="payload-pre-processing" containerID="cri-o://73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d" gracePeriod=30 Sep 11 15:23:35.995406 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:35.995371 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r"] Sep 11 15:23:35.997386 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:35.997364 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c097d73-6v82r"] Sep 11 15:23:36.097413 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.097361 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:23:36.157984 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.157961 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd166718-2b5a-4b41-b9ff-d63faf789c0b-plugins-config-volume\") pod \"cd166718-2b5a-4b41-b9ff-d63faf789c0b\" (UID: \"cd166718-2b5a-4b41-b9ff-d63faf789c0b\") " Sep 11 15:23:36.158125 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.158071 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb4ct\" (UniqueName: \"kubernetes.io/projected/cd166718-2b5a-4b41-b9ff-d63faf789c0b-kube-api-access-bb4ct\") pod \"cd166718-2b5a-4b41-b9ff-d63faf789c0b\" (UID: \"cd166718-2b5a-4b41-b9ff-d63faf789c0b\") " Sep 11 15:23:36.158400 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.158372 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd166718-2b5a-4b41-b9ff-d63faf789c0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd166718-2b5a-4b41-b9ff-d63faf789c0b" (UID: "cd166718-2b5a-4b41-b9ff-d63faf789c0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:23:36.160471 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.160452 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd166718-2b5a-4b41-b9ff-d63faf789c0b-kube-api-access-bb4ct" (OuterVolumeSpecName: "kube-api-access-bb4ct") pod "cd166718-2b5a-4b41-b9ff-d63faf789c0b" (UID: "cd166718-2b5a-4b41-b9ff-d63faf789c0b"). InnerVolumeSpecName "kube-api-access-bb4ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:23:36.259546 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.259524 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd166718-2b5a-4b41-b9ff-d63faf789c0b-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:23:36.259546 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.259547 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb4ct\" (UniqueName: \"kubernetes.io/projected/cd166718-2b5a-4b41-b9ff-d63faf789c0b-kube-api-access-bb4ct\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:23:36.493024 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.492961 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6407a83e-51ba-41e9-9072-6db665379603" path="/var/lib/kubelet/pods/6407a83e-51ba-41e9-9072-6db665379603/volumes" Sep 11 15:23:36.677315 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.677284 2573 generic.go:356] "Generic (PLEG): container finished" podID="cd166718-2b5a-4b41-b9ff-d63faf789c0b" containerID="73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d" exitCode=0 Sep 11 15:23:36.677418 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.677323 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" Sep 11 15:23:36.677418 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.677372 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" event={"ID":"cd166718-2b5a-4b41-b9ff-d63faf789c0b","Type":"ContainerDied","Data":"73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d"} Sep 11 15:23:36.677511 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.677421 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2" event={"ID":"cd166718-2b5a-4b41-b9ff-d63faf789c0b","Type":"ContainerDied","Data":"41bf8767da822bdab7b0d51a2df3b171efe0afcbb9203b1cb0fa76b03e919b7d"} Sep 11 15:23:36.677511 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.677443 2573 scope.go:117] "RemoveContainer" containerID="73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d" Sep 11 15:23:36.686177 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.686158 2573 scope.go:117] "RemoveContainer" containerID="73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d" Sep 11 15:23:36.686469 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:23:36.686450 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d\": container with ID starting with 73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d not found: ID does not exist" containerID="73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d" Sep 11 15:23:36.686513 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.686477 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d"} err="failed to get container status \"73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d\": rpc error: code = NotFound desc = could not find container \"73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d\": container with ID starting with 73ac8537f26cf265fe784ca4359a5d83a361c02240440da8659dc63fc826183d not found: ID does not exist" Sep 11 15:23:36.693851 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.693826 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2"] Sep 11 15:23:36.695643 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:36.695625 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c097d73-6c7988df74-xxlj2"] Sep 11 15:23:38.436480 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:38.436452 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:23:38.437503 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:38.437479 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:23:38.493045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:38.493020 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd166718-2b5a-4b41-b9ff-d63faf789c0b" path="/var/lib/kubelet/pods/cd166718-2b5a-4b41-b9ff-d63faf789c0b/volumes" Sep 11 15:23:42.521230 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521193 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx"] Sep 11 15:23:42.521596 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521584 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53427c8-c6ef-479a-8da4-b9d3b022439b" containerName="revoke-keys" Sep 11 15:23:42.521656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521597 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53427c8-c6ef-479a-8da4-b9d3b022439b" containerName="revoke-keys" Sep 11 15:23:42.521656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521607 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4" containerName="payload-pre-processing" Sep 11 15:23:42.521656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521613 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4" containerName="payload-pre-processing" Sep 11 15:23:42.521656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521620 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd166718-2b5a-4b41-b9ff-d63faf789c0b" containerName="payload-pre-processing" Sep 11 15:23:42.521656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521626 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd166718-2b5a-4b41-b9ff-d63faf789c0b" containerName="payload-pre-processing" Sep 11 15:23:42.521656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521640 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab651a9-7564-4d22-9d39-462c40b179d2" containerName="payload-processing" Sep 11 15:23:42.521656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521645 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab651a9-7564-4d22-9d39-462c40b179d2" containerName="payload-processing" Sep 11 15:23:42.521656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521654 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6407a83e-51ba-41e9-9072-6db665379603" containerName="revoke-keys" Sep 11 15:23:42.521656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521659 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6407a83e-51ba-41e9-9072-6db665379603" containerName="revoke-keys" Sep 11 15:23:42.522102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521716 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6407a83e-51ba-41e9-9072-6db665379603" containerName="revoke-keys" Sep 11 15:23:42.522102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521723 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab651a9-7564-4d22-9d39-462c40b179d2" containerName="payload-processing" Sep 11 15:23:42.522102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521731 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd166718-2b5a-4b41-b9ff-d63faf789c0b" containerName="payload-pre-processing" Sep 11 15:23:42.522102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521736 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53427c8-c6ef-479a-8da4-b9d3b022439b" containerName="revoke-keys" Sep 11 15:23:42.522102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.521743 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="63cb2f2f-b3a0-429d-91bb-dbdfa30fd9e4" containerName="payload-pre-processing" Sep 11 15:23:42.526553 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.526534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:23:42.529868 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.529848 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-df3af038-dockercfg-fpvmk\"" Sep 11 15:23:42.529955 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.529884 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-df3af038\"" Sep 11 15:23:42.540901 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.540877 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx"] Sep 11 15:23:42.610989 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.610959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfm4d\" (UniqueName: \"kubernetes.io/projected/9581a0f9-8a87-4236-8a40-a349664ca949-kube-api-access-kfm4d\") pod \"payload-processing-e2e-derive-df3af038-595454895c-995sx\" (UID: \"9581a0f9-8a87-4236-8a40-a349664ca949\") " pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:23:42.611138 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.611086 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9581a0f9-8a87-4236-8a40-a349664ca949-plugins-config-volume\") pod \"payload-processing-e2e-derive-df3af038-595454895c-995sx\" (UID: \"9581a0f9-8a87-4236-8a40-a349664ca949\") " pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:23:42.711495 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.711466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9581a0f9-8a87-4236-8a40-a349664ca949-plugins-config-volume\") pod \"payload-processing-e2e-derive-df3af038-595454895c-995sx\" (UID: \"9581a0f9-8a87-4236-8a40-a349664ca949\") " pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:23:42.711602 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.711557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm4d\" (UniqueName: \"kubernetes.io/projected/9581a0f9-8a87-4236-8a40-a349664ca949-kube-api-access-kfm4d\") pod \"payload-processing-e2e-derive-df3af038-595454895c-995sx\" (UID: \"9581a0f9-8a87-4236-8a40-a349664ca949\") " pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:23:42.712240 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.712222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9581a0f9-8a87-4236-8a40-a349664ca949-plugins-config-volume\") pod \"payload-processing-e2e-derive-df3af038-595454895c-995sx\" (UID: \"9581a0f9-8a87-4236-8a40-a349664ca949\") " pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:23:42.719496 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.719469 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfm4d\" (UniqueName: \"kubernetes.io/projected/9581a0f9-8a87-4236-8a40-a349664ca949-kube-api-access-kfm4d\") pod \"payload-processing-e2e-derive-df3af038-595454895c-995sx\" (UID: \"9581a0f9-8a87-4236-8a40-a349664ca949\") " pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:23:42.836889 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:42.836863 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:23:43.109413 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:43.109387 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx"] Sep 11 15:23:43.111203 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:23:43.111174 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9581a0f9_8a87_4236_8a40_a349664ca949.slice/crio-ac54b18813d3be26dd24e6009db7e67277d7e510050e32c5c57c928b33768332 WatchSource:0}: Error finding container ac54b18813d3be26dd24e6009db7e67277d7e510050e32c5c57c928b33768332: Status 404 returned error can't find the container with id ac54b18813d3be26dd24e6009db7e67277d7e510050e32c5c57c928b33768332 Sep 11 15:23:43.713086 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:43.713055 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" event={"ID":"9581a0f9-8a87-4236-8a40-a349664ca949","Type":"ContainerStarted","Data":"1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db"} Sep 11 15:23:43.713443 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:43.713115 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:23:43.713443 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:43.713137 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" event={"ID":"9581a0f9-8a87-4236-8a40-a349664ca949","Type":"ContainerStarted","Data":"ac54b18813d3be26dd24e6009db7e67277d7e510050e32c5c57c928b33768332"} Sep 11 15:23:43.730174 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:43.730128 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" podStartSLOduration=1.7301150669999998 podStartE2EDuration="1.730115067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:23:43.729038186 +0000 UTC m=+905.774363166" watchObservedRunningTime="2026-09-11 15:23:43.730115067 +0000 UTC m=+905.775440044" Sep 11 15:23:54.720936 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:23:54.720906 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:24:04.979260 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:04.979225 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x"] Sep 11 15:24:04.982536 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:04.982521 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" Sep 11 15:24:04.985057 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:04.985034 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:24:04.985181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:04.985039 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:24:04.990522 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:04.990497 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x"] Sep 11 15:24:05.090469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:05.090442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4849a611-6398-48db-bfc1-9d6632205107-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df3af038-lzw7x\" (UID: \"4849a611-6398-48db-bfc1-9d6632205107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" Sep 11 15:24:05.191796 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:05.191743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4849a611-6398-48db-bfc1-9d6632205107-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df3af038-lzw7x\" (UID: \"4849a611-6398-48db-bfc1-9d6632205107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" Sep 11 15:24:05.192322 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:05.192302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4849a611-6398-48db-bfc1-9d6632205107-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df3af038-lzw7x\" (UID: \"4849a611-6398-48db-bfc1-9d6632205107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" Sep 11 15:24:05.294740 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:05.294673 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" Sep 11 15:24:05.569193 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:05.569167 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x"] Sep 11 15:24:05.571494 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:24:05.571468 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4849a611_6398_48db_bfc1_9d6632205107.slice/crio-0234c453c2929c6902ec7423634305a48e3cdc499b28663328958761159c3400 WatchSource:0}: Error finding container 0234c453c2929c6902ec7423634305a48e3cdc499b28663328958761159c3400: Status 404 returned error can't find the container with id 0234c453c2929c6902ec7423634305a48e3cdc499b28663328958761159c3400 Sep 11 15:24:05.831679 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:05.831649 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" event={"ID":"4849a611-6398-48db-bfc1-9d6632205107","Type":"ContainerStarted","Data":"3ccab57c688e33a0c67af8090f968118bc9ef8fee789ea9e0cf517300e5cf76b"} Sep 11 15:24:05.831812 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:05.831695 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" event={"ID":"4849a611-6398-48db-bfc1-9d6632205107","Type":"ContainerStarted","Data":"0234c453c2929c6902ec7423634305a48e3cdc499b28663328958761159c3400"} Sep 11 15:24:05.846718 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:05.846673 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" podStartSLOduration=1.8466568589999999 podStartE2EDuration="1.846656859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:24:05.846128209 +0000 UTC m=+927.891453190" watchObservedRunningTime="2026-09-11 15:24:05.846656859 +0000 UTC m=+927.891981851" Sep 11 15:24:06.838394 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:06.838362 2573 generic.go:356] "Generic (PLEG): container finished" podID="4849a611-6398-48db-bfc1-9d6632205107" containerID="3ccab57c688e33a0c67af8090f968118bc9ef8fee789ea9e0cf517300e5cf76b" exitCode=0 Sep 11 15:24:06.838843 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:06.838441 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" event={"ID":"4849a611-6398-48db-bfc1-9d6632205107","Type":"ContainerDied","Data":"3ccab57c688e33a0c67af8090f968118bc9ef8fee789ea9e0cf517300e5cf76b"} Sep 11 15:24:07.971904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:07.971877 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" Sep 11 15:24:08.114522 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:08.114451 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4849a611-6398-48db-bfc1-9d6632205107-maas-api-service-ca\") pod \"4849a611-6398-48db-bfc1-9d6632205107\" (UID: \"4849a611-6398-48db-bfc1-9d6632205107\") " Sep 11 15:24:08.114861 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:08.114836 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4849a611-6398-48db-bfc1-9d6632205107-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4849a611-6398-48db-bfc1-9d6632205107" (UID: "4849a611-6398-48db-bfc1-9d6632205107"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:24:08.216167 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:08.216136 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4849a611-6398-48db-bfc1-9d6632205107-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:08.850021 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:08.849991 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" Sep 11 15:24:08.850170 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:08.850029 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x" event={"ID":"4849a611-6398-48db-bfc1-9d6632205107","Type":"ContainerDied","Data":"0234c453c2929c6902ec7423634305a48e3cdc499b28663328958761159c3400"} Sep 11 15:24:08.850170 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:08.850073 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0234c453c2929c6902ec7423634305a48e3cdc499b28663328958761159c3400" Sep 11 15:24:12.328111 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.328076 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx"] Sep 11 15:24:12.328565 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.328331 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" podUID="9581a0f9-8a87-4236-8a40-a349664ca949" containerName="payload-processing" containerID="cri-o://1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db" gracePeriod=30 Sep 11 15:24:12.494159 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.494133 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x"] Sep 11 15:24:12.494309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.494171 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df3af038-lzw7x"] Sep 11 15:24:12.585795 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.585714 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:24:12.757893 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.757862 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfm4d\" (UniqueName: \"kubernetes.io/projected/9581a0f9-8a87-4236-8a40-a349664ca949-kube-api-access-kfm4d\") pod \"9581a0f9-8a87-4236-8a40-a349664ca949\" (UID: \"9581a0f9-8a87-4236-8a40-a349664ca949\") " Sep 11 15:24:12.758028 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.757906 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9581a0f9-8a87-4236-8a40-a349664ca949-plugins-config-volume\") pod \"9581a0f9-8a87-4236-8a40-a349664ca949\" (UID: \"9581a0f9-8a87-4236-8a40-a349664ca949\") " Sep 11 15:24:12.758340 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.758312 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9581a0f9-8a87-4236-8a40-a349664ca949-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9581a0f9-8a87-4236-8a40-a349664ca949" (UID: "9581a0f9-8a87-4236-8a40-a349664ca949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:24:12.760066 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.760044 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9581a0f9-8a87-4236-8a40-a349664ca949-kube-api-access-kfm4d" (OuterVolumeSpecName: "kube-api-access-kfm4d") pod "9581a0f9-8a87-4236-8a40-a349664ca949" (UID: "9581a0f9-8a87-4236-8a40-a349664ca949"). InnerVolumeSpecName "kube-api-access-kfm4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:24:12.859095 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.859036 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfm4d\" (UniqueName: \"kubernetes.io/projected/9581a0f9-8a87-4236-8a40-a349664ca949-kube-api-access-kfm4d\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:12.859095 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.859060 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9581a0f9-8a87-4236-8a40-a349664ca949-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:12.871505 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.871476 2573 generic.go:356] "Generic (PLEG): container finished" podID="9581a0f9-8a87-4236-8a40-a349664ca949" containerID="1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db" exitCode=0 Sep 11 15:24:12.871598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.871520 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" Sep 11 15:24:12.871598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.871566 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" event={"ID":"9581a0f9-8a87-4236-8a40-a349664ca949","Type":"ContainerDied","Data":"1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db"} Sep 11 15:24:12.871676 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.871607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx" event={"ID":"9581a0f9-8a87-4236-8a40-a349664ca949","Type":"ContainerDied","Data":"ac54b18813d3be26dd24e6009db7e67277d7e510050e32c5c57c928b33768332"} Sep 11 15:24:12.871676 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.871629 2573 scope.go:117] "RemoveContainer" containerID="1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db" Sep 11 15:24:12.880293 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.880275 2573 scope.go:117] "RemoveContainer" containerID="1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db" Sep 11 15:24:12.880566 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:24:12.880548 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db\": container with ID starting with 1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db not found: ID does not exist" containerID="1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db" Sep 11 15:24:12.880615 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.880575 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db"} err="failed to get container status \"1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db\": rpc error: code = NotFound desc = could not find container \"1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db\": container with ID starting with 1bb4d83b49fbff70fc758432756d6aefe0cde46ba977e8195274fe9c158f61db not found: ID does not exist" Sep 11 15:24:12.891884 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.891864 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx"] Sep 11 15:24:12.895446 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:12.895427 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-df3af038-595454895c-995sx"] Sep 11 15:24:14.492485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:14.492454 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4849a611-6398-48db-bfc1-9d6632205107" path="/var/lib/kubelet/pods/4849a611-6398-48db-bfc1-9d6632205107/volumes" Sep 11 15:24:14.492871 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:14.492815 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9581a0f9-8a87-4236-8a40-a349664ca949" path="/var/lib/kubelet/pods/9581a0f9-8a87-4236-8a40-a349664ca949/volumes" Sep 11 15:24:20.272001 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.271970 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz"] Sep 11 15:24:20.272356 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.272341 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9581a0f9-8a87-4236-8a40-a349664ca949" containerName="payload-processing" Sep 11 15:24:20.272396 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.272357 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9581a0f9-8a87-4236-8a40-a349664ca949" containerName="payload-processing" Sep 11 15:24:20.272396 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.272376 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4849a611-6398-48db-bfc1-9d6632205107" containerName="revoke-keys" Sep 11 15:24:20.272396 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.272381 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4849a611-6398-48db-bfc1-9d6632205107" containerName="revoke-keys" Sep 11 15:24:20.272493 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.272442 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9581a0f9-8a87-4236-8a40-a349664ca949" containerName="payload-processing" Sep 11 15:24:20.272493 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.272451 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4849a611-6398-48db-bfc1-9d6632205107" containerName="revoke-keys" Sep 11 15:24:20.276940 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.276920 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.279385 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.279365 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-91304794-kserve-self-signed-certs\"" Sep 11 15:24:20.291052 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.286132 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz"] Sep 11 15:24:20.420958 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.420929 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-home\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.421065 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.420970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-model-cache\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.421065 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.421042 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-dshm\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.421147 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.421113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-tmp-dir\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.421192 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.421149 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a1f175-e843-4a77-a3af-dd19936450a0-tls-certs\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.421192 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.421182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkcrt\" (UniqueName: \"kubernetes.io/projected/61a1f175-e843-4a77-a3af-dd19936450a0-kube-api-access-xkcrt\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.521750 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.521719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-model-cache\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.521883 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.521817 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-dshm\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.521883 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.521846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-tmp-dir\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.521883 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.521881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a1f175-e843-4a77-a3af-dd19936450a0-tls-certs\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.522073 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.521916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkcrt\" (UniqueName: \"kubernetes.io/projected/61a1f175-e843-4a77-a3af-dd19936450a0-kube-api-access-xkcrt\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.522073 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.521959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-home\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.522228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.522187 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-model-cache\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.522294 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.522261 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-tmp-dir\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.522346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.522328 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-home\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.524483 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.524457 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-dshm\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.524723 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.524703 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a1f175-e843-4a77-a3af-dd19936450a0-tls-certs\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.530501 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.530478 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkcrt\" (UniqueName: \"kubernetes.io/projected/61a1f175-e843-4a77-a3af-dd19936450a0-kube-api-access-xkcrt\") pod \"e2e-conflict-b-91304794-kserve-58945dd87b-nckjz\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.592262 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.592242 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:20.867356 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.867330 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz"] Sep 11 15:24:20.870000 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:24:20.869970 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61a1f175_e843_4a77_a3af_dd19936450a0.slice/crio-6f084c537bcd4211ddc4a5e01d389077a2fd4477bd3d3e739fa7d797f64134d3 WatchSource:0}: Error finding container 6f084c537bcd4211ddc4a5e01d389077a2fd4477bd3d3e739fa7d797f64134d3: Status 404 returned error can't find the container with id 6f084c537bcd4211ddc4a5e01d389077a2fd4477bd3d3e739fa7d797f64134d3 Sep 11 15:24:20.914428 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:20.914401 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" event={"ID":"61a1f175-e843-4a77-a3af-dd19936450a0","Type":"ContainerStarted","Data":"6f084c537bcd4211ddc4a5e01d389077a2fd4477bd3d3e739fa7d797f64134d3"} Sep 11 15:24:21.921659 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:21.921621 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" event={"ID":"61a1f175-e843-4a77-a3af-dd19936450a0","Type":"ContainerStarted","Data":"ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0"} Sep 11 15:24:21.940907 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:21.940862 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" podStartSLOduration=1.9408475250000001 podStartE2EDuration="1.940847525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:24:21.939952189 +0000 UTC m=+943.985277182" watchObservedRunningTime="2026-09-11 15:24:21.940847525 +0000 UTC m=+943.986172504" Sep 11 15:24:30.593331 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:30.593295 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:30.593331 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:30.593342 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:30.605878 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:30.605855 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:30.979847 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:30.979740 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:36.968455 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:36.968423 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd"] Sep 11 15:24:36.972139 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:36.972116 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:24:36.974621 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:36.974595 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-ca78ef-dockercfg-lnwwj\"" Sep 11 15:24:36.974753 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:36.974645 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-ca78ef\"" Sep 11 15:24:36.984097 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:36.984072 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd"] Sep 11 15:24:36.997946 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:36.997924 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz"] Sep 11 15:24:37.053941 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:37.053911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjg2\" (UniqueName: \"kubernetes.io/projected/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-kube-api-access-whjg2\") pod \"payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd\" (UID: \"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530\") " pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:24:37.054078 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:37.054054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd\" (UID: \"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530\") " pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:24:37.155283 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:37.155238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjg2\" (UniqueName: \"kubernetes.io/projected/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-kube-api-access-whjg2\") pod \"payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd\" (UID: \"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530\") " pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:24:37.155425 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:37.155338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd\" (UID: \"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530\") " pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:24:37.156018 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:37.156001 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd\" (UID: \"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530\") " pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:24:37.162913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:37.162890 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjg2\" (UniqueName: \"kubernetes.io/projected/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-kube-api-access-whjg2\") pod \"payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd\" (UID: \"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530\") " pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:24:37.286300 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:37.286238 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:24:37.560852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:37.560822 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd"] Sep 11 15:24:38.005426 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:38.005396 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" event={"ID":"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530","Type":"ContainerStarted","Data":"6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0"} Sep 11 15:24:38.005832 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:38.005446 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" event={"ID":"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530","Type":"ContainerStarted","Data":"9be4b5a9d1f26791c9b105dadd44977ec40cb8f9de522a1e050f3115b785c38f"} Sep 11 15:24:38.005832 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:38.005557 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:24:38.023331 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:38.023282 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" podStartSLOduration=2.023265381 podStartE2EDuration="2.023265381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:24:38.02102469 +0000 UTC m=+960.066349671" watchObservedRunningTime="2026-09-11 15:24:38.023265381 +0000 UTC m=+960.068590362" Sep 11 15:24:40.567136 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:40.567104 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2"] Sep 11 15:24:49.013698 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:49.013656 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:24:52.018077 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.018039 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" podUID="61a1f175-e843-4a77-a3af-dd19936450a0" containerName="main" containerID="cri-o://ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0" gracePeriod=45 Sep 11 15:24:52.261610 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.261583 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:52.386474 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386447 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-tmp-dir\") pod \"61a1f175-e843-4a77-a3af-dd19936450a0\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " Sep 11 15:24:52.386609 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386488 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-dshm\") pod \"61a1f175-e843-4a77-a3af-dd19936450a0\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " Sep 11 15:24:52.386609 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386512 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-home\") pod \"61a1f175-e843-4a77-a3af-dd19936450a0\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " Sep 11 15:24:52.386609 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386539 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkcrt\" (UniqueName: \"kubernetes.io/projected/61a1f175-e843-4a77-a3af-dd19936450a0-kube-api-access-xkcrt\") pod \"61a1f175-e843-4a77-a3af-dd19936450a0\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " Sep 11 15:24:52.386609 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386577 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-model-cache\") pod \"61a1f175-e843-4a77-a3af-dd19936450a0\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " Sep 11 15:24:52.386865 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386625 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a1f175-e843-4a77-a3af-dd19936450a0-tls-certs\") pod \"61a1f175-e843-4a77-a3af-dd19936450a0\" (UID: \"61a1f175-e843-4a77-a3af-dd19936450a0\") " Sep 11 15:24:52.386865 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386687 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61a1f175-e843-4a77-a3af-dd19936450a0" (UID: "61a1f175-e843-4a77-a3af-dd19936450a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:52.386865 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386841 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-home" (OuterVolumeSpecName: "home") pod "61a1f175-e843-4a77-a3af-dd19936450a0" (UID: "61a1f175-e843-4a77-a3af-dd19936450a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:52.387031 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386884 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "61a1f175-e843-4a77-a3af-dd19936450a0" (UID: "61a1f175-e843-4a77-a3af-dd19936450a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:52.387031 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386898 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-tmp-dir\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:52.387031 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.386931 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-home\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:52.388857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.388833 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a1f175-e843-4a77-a3af-dd19936450a0-kube-api-access-xkcrt" (OuterVolumeSpecName: "kube-api-access-xkcrt") pod "61a1f175-e843-4a77-a3af-dd19936450a0" (UID: "61a1f175-e843-4a77-a3af-dd19936450a0"). InnerVolumeSpecName "kube-api-access-xkcrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:24:52.388857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.388846 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-dshm" (OuterVolumeSpecName: "dshm") pod "61a1f175-e843-4a77-a3af-dd19936450a0" (UID: "61a1f175-e843-4a77-a3af-dd19936450a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:52.388992 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.388903 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a1f175-e843-4a77-a3af-dd19936450a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61a1f175-e843-4a77-a3af-dd19936450a0" (UID: "61a1f175-e843-4a77-a3af-dd19936450a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:24:52.488174 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.488153 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkcrt\" (UniqueName: \"kubernetes.io/projected/61a1f175-e843-4a77-a3af-dd19936450a0-kube-api-access-xkcrt\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:52.488174 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.488173 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:52.488307 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.488182 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a1f175-e843-4a77-a3af-dd19936450a0-tls-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:52.488307 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:52.488190 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a1f175-e843-4a77-a3af-dd19936450a0-dshm\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:53.082932 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:53.082904 2573 generic.go:356] "Generic (PLEG): container finished" podID="61a1f175-e843-4a77-a3af-dd19936450a0" containerID="ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0" exitCode=0 Sep 11 15:24:53.083303 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:53.082957 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" Sep 11 15:24:53.083303 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:53.083001 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" event={"ID":"61a1f175-e843-4a77-a3af-dd19936450a0","Type":"ContainerDied","Data":"ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0"} Sep 11 15:24:53.083303 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:53.083040 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz" event={"ID":"61a1f175-e843-4a77-a3af-dd19936450a0","Type":"ContainerDied","Data":"6f084c537bcd4211ddc4a5e01d389077a2fd4477bd3d3e739fa7d797f64134d3"} Sep 11 15:24:53.083303 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:53.083057 2573 scope.go:117] "RemoveContainer" containerID="ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0" Sep 11 15:24:53.093989 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:53.093971 2573 scope.go:117] "RemoveContainer" containerID="ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0" Sep 11 15:24:53.094278 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:24:53.094260 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0\": container with ID starting with ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0 not found: ID does not exist" containerID="ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0" Sep 11 15:24:53.094352 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:53.094290 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0"} err="failed to get container status \"ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0\": rpc error: code = NotFound desc = could not find container \"ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0\": container with ID starting with ab589175fe490fd1cfc2ad1ca4bebd68ac6713a95a2b6e76c56a899d9b2da0a0 not found: ID does not exist" Sep 11 15:24:53.102003 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:53.101978 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz"] Sep 11 15:24:53.113300 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:53.113269 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-91304794-kserve-58945dd87b-nckjz"] Sep 11 15:24:54.492406 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:54.492371 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a1f175-e843-4a77-a3af-dd19936450a0" path="/var/lib/kubelet/pods/61a1f175-e843-4a77-a3af-dd19936450a0/volumes" Sep 11 15:24:55.586673 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.586624 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" podUID="9c8ed830-dffe-4e2a-ab54-3ef3201305a7" containerName="main" containerID="cri-o://c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73" gracePeriod=45 Sep 11 15:24:55.827041 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.827017 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:24:55.916576 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.916505 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-dshm\") pod \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " Sep 11 15:24:55.916576 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.916558 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzhjr\" (UniqueName: \"kubernetes.io/projected/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-kube-api-access-kzhjr\") pod \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " Sep 11 15:24:55.916794 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.916585 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-model-cache\") pod \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " Sep 11 15:24:55.916794 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.916625 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tmp-dir\") pod \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " Sep 11 15:24:55.916794 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.916657 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-home\") pod \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " Sep 11 15:24:55.916794 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.916694 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tls-certs\") pod \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\" (UID: \"9c8ed830-dffe-4e2a-ab54-3ef3201305a7\") " Sep 11 15:24:55.917012 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.916901 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c8ed830-dffe-4e2a-ab54-3ef3201305a7" (UID: "9c8ed830-dffe-4e2a-ab54-3ef3201305a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:55.917012 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.916915 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c8ed830-dffe-4e2a-ab54-3ef3201305a7" (UID: "9c8ed830-dffe-4e2a-ab54-3ef3201305a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:55.917012 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.916945 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-home" (OuterVolumeSpecName: "home") pod "9c8ed830-dffe-4e2a-ab54-3ef3201305a7" (UID: "9c8ed830-dffe-4e2a-ab54-3ef3201305a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:55.917153 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.917082 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:55.917153 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.917103 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tmp-dir\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:55.917153 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.917117 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-home\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:55.918830 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.918804 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-dshm" (OuterVolumeSpecName: "dshm") pod "9c8ed830-dffe-4e2a-ab54-3ef3201305a7" (UID: "9c8ed830-dffe-4e2a-ab54-3ef3201305a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:55.918942 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.918845 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-kube-api-access-kzhjr" (OuterVolumeSpecName: "kube-api-access-kzhjr") pod "9c8ed830-dffe-4e2a-ab54-3ef3201305a7" (UID: "9c8ed830-dffe-4e2a-ab54-3ef3201305a7"). InnerVolumeSpecName "kube-api-access-kzhjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:24:55.918942 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:55.918845 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c8ed830-dffe-4e2a-ab54-3ef3201305a7" (UID: "9c8ed830-dffe-4e2a-ab54-3ef3201305a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:24:56.017731 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.017705 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzhjr\" (UniqueName: \"kubernetes.io/projected/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-kube-api-access-kzhjr\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:56.017731 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.017729 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-tls-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:56.017888 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.017739 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8ed830-dffe-4e2a-ab54-3ef3201305a7-dshm\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:24:56.100577 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.100547 2573 generic.go:356] "Generic (PLEG): container finished" podID="9c8ed830-dffe-4e2a-ab54-3ef3201305a7" containerID="c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73" exitCode=0 Sep 11 15:24:56.100684 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.100586 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" Sep 11 15:24:56.100684 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.100636 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" event={"ID":"9c8ed830-dffe-4e2a-ab54-3ef3201305a7","Type":"ContainerDied","Data":"c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73"} Sep 11 15:24:56.100684 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.100679 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2" event={"ID":"9c8ed830-dffe-4e2a-ab54-3ef3201305a7","Type":"ContainerDied","Data":"a852e4479d8a88785193c0c571186561ccff1cb9674e4e303847f8f1891b6018"} Sep 11 15:24:56.100843 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.100697 2573 scope.go:117] "RemoveContainer" containerID="c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73" Sep 11 15:24:56.110636 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.110618 2573 scope.go:117] "RemoveContainer" containerID="c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73" Sep 11 15:24:56.110921 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:24:56.110905 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73\": container with ID starting with c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73 not found: ID does not exist" containerID="c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73" Sep 11 15:24:56.110973 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.110928 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73"} err="failed to get container status \"c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73\": rpc error: code = NotFound desc = could not find container \"c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73\": container with ID starting with c8a905b586a780d4a3de6d95593eb8ac1e45efb6faa59ff0669196b3f0384f73 not found: ID does not exist" Sep 11 15:24:56.121480 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.121459 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2"] Sep 11 15:24:56.127707 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.127686 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-91304794-kserve-bd4c5dfd6-dhsb2"] Sep 11 15:24:56.492805 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:24:56.492760 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c8ed830-dffe-4e2a-ab54-3ef3201305a7" path="/var/lib/kubelet/pods/9c8ed830-dffe-4e2a-ab54-3ef3201305a7/volumes" Sep 11 15:25:06.280446 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.280366 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb"] Sep 11 15:25:06.280850 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.280831 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a1f175-e843-4a77-a3af-dd19936450a0" containerName="main" Sep 11 15:25:06.280850 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.280846 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a1f175-e843-4a77-a3af-dd19936450a0" containerName="main" Sep 11 15:25:06.280944 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.280858 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c8ed830-dffe-4e2a-ab54-3ef3201305a7" containerName="main" Sep 11 15:25:06.280944 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.280863 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c8ed830-dffe-4e2a-ab54-3ef3201305a7" containerName="main" Sep 11 15:25:06.280944 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.280922 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c8ed830-dffe-4e2a-ab54-3ef3201305a7" containerName="main" Sep 11 15:25:06.280944 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.280929 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a1f175-e843-4a77-a3af-dd19936450a0" containerName="main" Sep 11 15:25:06.283815 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.283797 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.286962 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.286938 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ca78ef\"/\"openshift-service-ca.crt\"" Sep 11 15:25:06.287105 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.287068 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ca78ef\"/\"explicit-model-ca78ef-kserve-self-signed-certs\"" Sep 11 15:25:06.287105 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.286938 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ca78ef\"/\"default-dockercfg-4tlrs\"" Sep 11 15:25:06.287224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.287186 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ca78ef\"/\"kube-root-ca.crt\"" Sep 11 15:25:06.292096 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.292050 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb"] Sep 11 15:25:06.407304 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.407275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.407428 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.407312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-model-cache\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.407428 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.407338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-tmp-dir\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.407428 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.407411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-home\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.407534 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.407448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-dshm\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.407534 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.407476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gffn7\" (UniqueName: \"kubernetes.io/projected/fce72580-b0fd-4ace-9803-441b70e212c7-kube-api-access-gffn7\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.508618 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.508592 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-tmp-dir\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.508727 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.508638 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-home\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.508727 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.508692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-dshm\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.508837 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.508733 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gffn7\" (UniqueName: \"kubernetes.io/projected/fce72580-b0fd-4ace-9803-441b70e212c7-kube-api-access-gffn7\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.508837 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.508814 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.508932 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.508839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-model-cache\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.509049 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.509030 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-tmp-dir\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.509116 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.509052 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-home\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.509177 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.509122 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-model-cache\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.511038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.511011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-dshm\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.511410 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.511387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.517058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.517041 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gffn7\" (UniqueName: \"kubernetes.io/projected/fce72580-b0fd-4ace-9803-441b70e212c7-kube-api-access-gffn7\") pod \"explicit-model-ca78ef-kserve-78d76fcb77-8sbpb\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.605089 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.605065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:06.878480 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:06.878451 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb"] Sep 11 15:25:06.880087 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:25:06.880051 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfce72580_b0fd_4ace_9803_441b70e212c7.slice/crio-c2a7ba5636e50b9202c9c1bcc28ccdad2efd3bc789393d38b85203832d671442 WatchSource:0}: Error finding container c2a7ba5636e50b9202c9c1bcc28ccdad2efd3bc789393d38b85203832d671442: Status 404 returned error can't find the container with id c2a7ba5636e50b9202c9c1bcc28ccdad2efd3bc789393d38b85203832d671442 Sep 11 15:25:07.157905 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:07.157822 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" event={"ID":"fce72580-b0fd-4ace-9803-441b70e212c7","Type":"ContainerStarted","Data":"1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5"} Sep 11 15:25:07.157905 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:07.157860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" event={"ID":"fce72580-b0fd-4ace-9803-441b70e212c7","Type":"ContainerStarted","Data":"c2a7ba5636e50b9202c9c1bcc28ccdad2efd3bc789393d38b85203832d671442"} Sep 11 15:25:07.175034 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:07.174982 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" podStartSLOduration=1.174966847 podStartE2EDuration="1.174966847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:25:07.172599851 +0000 UTC m=+989.217924856" watchObservedRunningTime="2026-09-11 15:25:07.174966847 +0000 UTC m=+989.220291845" Sep 11 15:25:16.605845 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:16.605808 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:16.606310 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:16.605858 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:16.618608 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:16.618585 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:17.222815 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:17.222762 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:17.728570 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:17.728538 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb"] Sep 11 15:25:17.854703 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:17.854668 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc"] Sep 11 15:25:17.858562 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:17.858543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" Sep 11 15:25:17.860795 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:17.860742 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:25:17.860921 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:17.860742 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:25:17.864625 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:17.864602 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc"] Sep 11 15:25:17.907444 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:17.907418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ae671-2912-40a2-81f2-357677e2756e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc\" (UID: \"5a4ae671-2912-40a2-81f2-357677e2756e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" Sep 11 15:25:18.008125 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:18.008066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ae671-2912-40a2-81f2-357677e2756e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc\" (UID: \"5a4ae671-2912-40a2-81f2-357677e2756e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" Sep 11 15:25:18.008631 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:18.008611 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ae671-2912-40a2-81f2-357677e2756e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc\" (UID: \"5a4ae671-2912-40a2-81f2-357677e2756e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" Sep 11 15:25:18.196627 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:18.196591 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" Sep 11 15:25:18.311346 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:18.311281 2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-self-signed-certs: secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:18.311462 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:18.311368 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs podName:fce72580-b0fd-4ace-9803-441b70e212c7 nodeName:}" failed. No retries permitted until 2026-09-11 15:25:18.811345288 +0000 UTC m=+1000.856670262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs") pod "explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7") : secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:18.467164 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:18.467141 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc"] Sep 11 15:25:18.470072 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:25:18.470045 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4ae671_2912_40a2_81f2_357677e2756e.slice/crio-6de6a95ea9bb62a7a09564a9c0f7ec2f15bf0fd8b40028168ed07be1c492e766 WatchSource:0}: Error finding container 6de6a95ea9bb62a7a09564a9c0f7ec2f15bf0fd8b40028168ed07be1c492e766: Status 404 returned error can't find the container with id 6de6a95ea9bb62a7a09564a9c0f7ec2f15bf0fd8b40028168ed07be1c492e766 Sep 11 15:25:18.815933 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:18.815909 2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-self-signed-certs: secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:18.816224 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:18.815975 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs podName:fce72580-b0fd-4ace-9803-441b70e212c7 nodeName:}" failed. No retries permitted until 2026-09-11 15:25:19.815957673 +0000 UTC m=+1001.861282631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs") pod "explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7") : secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:19.225583 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:19.225547 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" event={"ID":"5a4ae671-2912-40a2-81f2-357677e2756e","Type":"ContainerStarted","Data":"9e7520361dfa90bca1970752f0fe3032195cfb0ebb23a285c2c27407df2c0259"} Sep 11 15:25:19.225742 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:19.225608 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" event={"ID":"5a4ae671-2912-40a2-81f2-357677e2756e","Type":"ContainerStarted","Data":"6de6a95ea9bb62a7a09564a9c0f7ec2f15bf0fd8b40028168ed07be1c492e766"} Sep 11 15:25:19.247576 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:19.246768 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" podStartSLOduration=2.2467498949999998 podStartE2EDuration="2.246749895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:25:19.239263435 +0000 UTC m=+1001.284588416" watchObservedRunningTime="2026-09-11 15:25:19.246749895 +0000 UTC m=+1001.292074875" Sep 11 15:25:19.823359 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:19.823327 2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-self-signed-certs: secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:19.823732 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:19.823407 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs podName:fce72580-b0fd-4ace-9803-441b70e212c7 nodeName:}" failed. No retries permitted until 2026-09-11 15:25:21.82339226 +0000 UTC m=+1003.868717218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs") pod "explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7") : secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:20.232465 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:20.232433 2573 generic.go:356] "Generic (PLEG): container finished" podID="5a4ae671-2912-40a2-81f2-357677e2756e" containerID="9e7520361dfa90bca1970752f0fe3032195cfb0ebb23a285c2c27407df2c0259" exitCode=0 Sep 11 15:25:20.232627 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:20.232502 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" event={"ID":"5a4ae671-2912-40a2-81f2-357677e2756e","Type":"ContainerDied","Data":"9e7520361dfa90bca1970752f0fe3032195cfb0ebb23a285c2c27407df2c0259"} Sep 11 15:25:21.371742 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:21.371715 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" Sep 11 15:25:21.435462 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:21.435435 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ae671-2912-40a2-81f2-357677e2756e-maas-api-service-ca\") pod \"5a4ae671-2912-40a2-81f2-357677e2756e\" (UID: \"5a4ae671-2912-40a2-81f2-357677e2756e\") " Sep 11 15:25:21.435744 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:21.435721 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4ae671-2912-40a2-81f2-357677e2756e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a4ae671-2912-40a2-81f2-357677e2756e" (UID: "5a4ae671-2912-40a2-81f2-357677e2756e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:25:21.537043 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:21.536763 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ae671-2912-40a2-81f2-357677e2756e-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:25:21.838687 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:21.838661 2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-self-signed-certs: secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:21.838849 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:21.838724 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs podName:fce72580-b0fd-4ace-9803-441b70e212c7 nodeName:}" failed. No retries permitted until 2026-09-11 15:25:25.838709607 +0000 UTC m=+1007.884034565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs") pod "explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7") : secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:22.244412 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:22.244334 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" Sep 11 15:25:22.244412 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:22.244360 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc" event={"ID":"5a4ae671-2912-40a2-81f2-357677e2756e","Type":"ContainerDied","Data":"6de6a95ea9bb62a7a09564a9c0f7ec2f15bf0fd8b40028168ed07be1c492e766"} Sep 11 15:25:22.244412 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:22.244405 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6de6a95ea9bb62a7a09564a9c0f7ec2f15bf0fd8b40028168ed07be1c492e766" Sep 11 15:25:25.873884 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:25.873844 2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-self-signed-certs: secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:25.874369 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:25.873938 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs podName:fce72580-b0fd-4ace-9803-441b70e212c7 nodeName:}" failed. No retries permitted until 2026-09-11 15:25:33.873917852 +0000 UTC m=+1015.919242810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs") pod "explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7") : secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:28.411575 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.411540 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd"] Sep 11 15:25:28.411950 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.411813 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" podUID="ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530" containerName="payload-processing" containerID="cri-o://6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0" gracePeriod=30 Sep 11 15:25:28.577730 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.577699 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc"] Sep 11 15:25:28.579375 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.579353 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ca78ef-hhrsc"] Sep 11 15:25:28.678744 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.678721 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:25:28.797025 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.796998 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whjg2\" (UniqueName: \"kubernetes.io/projected/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-kube-api-access-whjg2\") pod \"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530\" (UID: \"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530\") " Sep 11 15:25:28.797159 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.797058 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-plugins-config-volume\") pod \"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530\" (UID: \"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530\") " Sep 11 15:25:28.797444 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.797422 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530" (UID: "ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:25:28.799096 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.799073 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-kube-api-access-whjg2" (OuterVolumeSpecName: "kube-api-access-whjg2") pod "ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530" (UID: "ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530"). InnerVolumeSpecName "kube-api-access-whjg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:25:28.897759 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.897715 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whjg2\" (UniqueName: \"kubernetes.io/projected/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-kube-api-access-whjg2\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:25:28.897759 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:28.897757 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:25:29.280764 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:29.280726 2573 generic.go:356] "Generic (PLEG): container finished" podID="ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530" containerID="6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0" exitCode=0 Sep 11 15:25:29.280925 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:29.280798 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" Sep 11 15:25:29.280925 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:29.280792 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" event={"ID":"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530","Type":"ContainerDied","Data":"6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0"} Sep 11 15:25:29.280925 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:29.280913 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd" event={"ID":"ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530","Type":"ContainerDied","Data":"9be4b5a9d1f26791c9b105dadd44977ec40cb8f9de522a1e050f3115b785c38f"} Sep 11 15:25:29.281085 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:29.280937 2573 scope.go:117] "RemoveContainer" containerID="6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0" Sep 11 15:25:29.290051 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:29.290031 2573 scope.go:117] "RemoveContainer" containerID="6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0" Sep 11 15:25:29.290311 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:29.290292 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0\": container with ID starting with 6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0 not found: ID does not exist" containerID="6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0" Sep 11 15:25:29.290379 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:29.290320 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0"} err="failed to get container status \"6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0\": rpc error: code = NotFound desc = could not find container \"6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0\": container with ID starting with 6a3ef187d61438518b7636a59c3fd43da258a059f7e289c73a07670298997db0 not found: ID does not exist" Sep 11 15:25:29.302071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:29.302047 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd"] Sep 11 15:25:29.307481 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:29.307463 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ca78ef-57877cd4b9-sj6gd"] Sep 11 15:25:30.493133 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:30.493094 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4ae671-2912-40a2-81f2-357677e2756e" path="/var/lib/kubelet/pods/5a4ae671-2912-40a2-81f2-357677e2756e/volumes" Sep 11 15:25:30.493501 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:30.493438 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530" path="/var/lib/kubelet/pods/ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530/volumes" Sep 11 15:25:33.943577 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:33.943543 2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-self-signed-certs: secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:33.943969 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:33.943614 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs podName:fce72580-b0fd-4ace-9803-441b70e212c7 nodeName:}" failed. No retries permitted until 2026-09-11 15:25:49.943597077 +0000 UTC m=+1031.988922039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs") pod "explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7") : secret "explicit-model-ca78ef-kserve-self-signed-certs" not found Sep 11 15:25:34.246902 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.246827 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" podUID="fce72580-b0fd-4ace-9803-441b70e212c7" containerName="main" containerID="cri-o://1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5" gracePeriod=45 Sep 11 15:25:34.496955 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.496894 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:34.549513 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.549490 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gffn7\" (UniqueName: \"kubernetes.io/projected/fce72580-b0fd-4ace-9803-441b70e212c7-kube-api-access-gffn7\") pod \"fce72580-b0fd-4ace-9803-441b70e212c7\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " Sep 11 15:25:34.549663 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.549536 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs\") pod \"fce72580-b0fd-4ace-9803-441b70e212c7\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " Sep 11 15:25:34.549663 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.549569 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-model-cache\") pod \"fce72580-b0fd-4ace-9803-441b70e212c7\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " Sep 11 15:25:34.549663 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.549591 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-home\") pod \"fce72580-b0fd-4ace-9803-441b70e212c7\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " Sep 11 15:25:34.549663 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.549623 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-tmp-dir\") pod \"fce72580-b0fd-4ace-9803-441b70e212c7\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " Sep 11 15:25:34.549925 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.549703 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-dshm\") pod \"fce72580-b0fd-4ace-9803-441b70e212c7\" (UID: \"fce72580-b0fd-4ace-9803-441b70e212c7\") " Sep 11 15:25:34.549925 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.549882 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-home" (OuterVolumeSpecName: "home") pod "fce72580-b0fd-4ace-9803-441b70e212c7" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:25:34.549925 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.549913 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "fce72580-b0fd-4ace-9803-441b70e212c7" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:25:34.550105 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.549969 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fce72580-b0fd-4ace-9803-441b70e212c7" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:25:34.550210 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.550191 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:25:34.550279 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.550213 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-home\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:25:34.550279 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.550227 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-tmp-dir\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:25:34.551974 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.551944 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce72580-b0fd-4ace-9803-441b70e212c7-kube-api-access-gffn7" (OuterVolumeSpecName: "kube-api-access-gffn7") pod "fce72580-b0fd-4ace-9803-441b70e212c7" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7"). InnerVolumeSpecName "kube-api-access-gffn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:25:34.552473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.552452 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fce72580-b0fd-4ace-9803-441b70e212c7" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:25:34.552556 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.552452 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-dshm" (OuterVolumeSpecName: "dshm") pod "fce72580-b0fd-4ace-9803-441b70e212c7" (UID: "fce72580-b0fd-4ace-9803-441b70e212c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:25:34.650644 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.650620 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gffn7\" (UniqueName: \"kubernetes.io/projected/fce72580-b0fd-4ace-9803-441b70e212c7-kube-api-access-gffn7\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:25:34.650644 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.650644 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce72580-b0fd-4ace-9803-441b70e212c7-tls-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:25:34.650827 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:34.650654 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce72580-b0fd-4ace-9803-441b70e212c7-dshm\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:25:35.313168 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:35.313132 2573 generic.go:356] "Generic (PLEG): container finished" podID="fce72580-b0fd-4ace-9803-441b70e212c7" containerID="1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5" exitCode=0 Sep 11 15:25:35.313524 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:35.313178 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" Sep 11 15:25:35.313524 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:35.313218 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" event={"ID":"fce72580-b0fd-4ace-9803-441b70e212c7","Type":"ContainerDied","Data":"1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5"} Sep 11 15:25:35.313524 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:35.313261 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb" event={"ID":"fce72580-b0fd-4ace-9803-441b70e212c7","Type":"ContainerDied","Data":"c2a7ba5636e50b9202c9c1bcc28ccdad2efd3bc789393d38b85203832d671442"} Sep 11 15:25:35.313524 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:35.313278 2573 scope.go:117] "RemoveContainer" containerID="1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5" Sep 11 15:25:35.323064 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:35.323047 2573 scope.go:117] "RemoveContainer" containerID="1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5" Sep 11 15:25:35.323333 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:25:35.323315 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5\": container with ID starting with 1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5 not found: ID does not exist" containerID="1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5" Sep 11 15:25:35.323385 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:35.323341 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5"} err="failed to get container status \"1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5\": rpc error: code = NotFound desc = could not find container \"1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5\": container with ID starting with 1905e2d1c01705442fe6e00cd46251c640aee95f38f18233b91b738e5f92aac5 not found: ID does not exist" Sep 11 15:25:35.334487 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:35.334461 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb"] Sep 11 15:25:35.337962 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:35.337940 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-ca78ef/explicit-model-ca78ef-kserve-78d76fcb77-8sbpb"] Sep 11 15:25:36.492362 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:36.492331 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce72580-b0fd-4ace-9803-441b70e212c7" path="/var/lib/kubelet/pods/fce72580-b0fd-4ace-9803-441b70e212c7/volumes" Sep 11 15:25:55.110314 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.110276 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt"] Sep 11 15:25:55.110929 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.110909 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530" containerName="payload-processing" Sep 11 15:25:55.110980 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.110933 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530" containerName="payload-processing" Sep 11 15:25:55.110980 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.110974 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4ae671-2912-40a2-81f2-357677e2756e" containerName="revoke-keys" Sep 11 15:25:55.111044 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.110983 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4ae671-2912-40a2-81f2-357677e2756e" containerName="revoke-keys" Sep 11 15:25:55.111044 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.110999 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce72580-b0fd-4ace-9803-441b70e212c7" containerName="main" Sep 11 15:25:55.111044 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.111008 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce72580-b0fd-4ace-9803-441b70e212c7" containerName="main" Sep 11 15:25:55.111152 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.111107 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3e6c07-f7f2-4cbd-a3aa-a754b55cf530" containerName="payload-processing" Sep 11 15:25:55.111152 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.111120 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce72580-b0fd-4ace-9803-441b70e212c7" containerName="main" Sep 11 15:25:55.111152 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.111129 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4ae671-2912-40a2-81f2-357677e2756e" containerName="revoke-keys" Sep 11 15:25:55.115971 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.115952 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44"] Sep 11 15:25:55.116081 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.116064 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:25:55.118429 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.118410 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0dd91403\"" Sep 11 15:25:55.119136 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.119123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:25:55.128346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.128323 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt"] Sep 11 15:25:55.130043 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.130020 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44"] Sep 11 15:25:55.214821 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.214793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5b62\" (UniqueName: \"kubernetes.io/projected/d3befa73-ddfc-4363-b5dc-450fe17bf51a-kube-api-access-g5b62\") pod \"payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44\" (UID: \"d3befa73-ddfc-4363-b5dc-450fe17bf51a\") " pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:25:55.214948 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.214858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmld\" (UniqueName: \"kubernetes.io/projected/7eda975c-26f7-415c-9efe-508b99368610-kube-api-access-rhmld\") pod \"payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt\" (UID: \"7eda975c-26f7-415c-9efe-508b99368610\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:25:55.214948 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.214929 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3befa73-ddfc-4363-b5dc-450fe17bf51a-plugins-config-volume\") pod \"payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44\" (UID: \"d3befa73-ddfc-4363-b5dc-450fe17bf51a\") " pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:25:55.215051 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.214963 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eda975c-26f7-415c-9efe-508b99368610-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt\" (UID: \"7eda975c-26f7-415c-9efe-508b99368610\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:25:55.315625 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.315601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmld\" (UniqueName: \"kubernetes.io/projected/7eda975c-26f7-415c-9efe-508b99368610-kube-api-access-rhmld\") pod \"payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt\" (UID: \"7eda975c-26f7-415c-9efe-508b99368610\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:25:55.315746 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.315636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3befa73-ddfc-4363-b5dc-450fe17bf51a-plugins-config-volume\") pod \"payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44\" (UID: \"d3befa73-ddfc-4363-b5dc-450fe17bf51a\") " pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:25:55.315746 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.315655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eda975c-26f7-415c-9efe-508b99368610-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt\" (UID: \"7eda975c-26f7-415c-9efe-508b99368610\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:25:55.315746 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.315701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5b62\" (UniqueName: \"kubernetes.io/projected/d3befa73-ddfc-4363-b5dc-450fe17bf51a-kube-api-access-g5b62\") pod \"payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44\" (UID: \"d3befa73-ddfc-4363-b5dc-450fe17bf51a\") " pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:25:55.316419 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.316393 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eda975c-26f7-415c-9efe-508b99368610-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt\" (UID: \"7eda975c-26f7-415c-9efe-508b99368610\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:25:55.316507 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.316419 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3befa73-ddfc-4363-b5dc-450fe17bf51a-plugins-config-volume\") pod \"payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44\" (UID: \"d3befa73-ddfc-4363-b5dc-450fe17bf51a\") " pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:25:55.323465 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.323443 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmld\" (UniqueName: \"kubernetes.io/projected/7eda975c-26f7-415c-9efe-508b99368610-kube-api-access-rhmld\") pod \"payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt\" (UID: \"7eda975c-26f7-415c-9efe-508b99368610\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:25:55.324087 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.324067 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5b62\" (UniqueName: \"kubernetes.io/projected/d3befa73-ddfc-4363-b5dc-450fe17bf51a-kube-api-access-g5b62\") pod \"payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44\" (UID: \"d3befa73-ddfc-4363-b5dc-450fe17bf51a\") " pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:25:55.427968 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.427911 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:25:55.435287 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.435271 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:25:55.701656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.701631 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt"] Sep 11 15:25:55.704331 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:25:55.704297 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eda975c_26f7_415c_9efe_508b99368610.slice/crio-1316b1a24578ef3b3b81e45445ae0a594d321e0f59bf5eaf99cc53ef5f4fce5b WatchSource:0}: Error finding container 1316b1a24578ef3b3b81e45445ae0a594d321e0f59bf5eaf99cc53ef5f4fce5b: Status 404 returned error can't find the container with id 1316b1a24578ef3b3b81e45445ae0a594d321e0f59bf5eaf99cc53ef5f4fce5b Sep 11 15:25:55.706075 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:55.706057 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44"] Sep 11 15:25:55.707688 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:25:55.707616 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3befa73_ddfc_4363_b5dc_450fe17bf51a.slice/crio-59a9bbe6e9026522bb51688518b317b753324c8bbaa3273c71e4dca95126af49 WatchSource:0}: Error finding container 59a9bbe6e9026522bb51688518b317b753324c8bbaa3273c71e4dca95126af49: Status 404 returned error can't find the container with id 59a9bbe6e9026522bb51688518b317b753324c8bbaa3273c71e4dca95126af49 Sep 11 15:25:56.433210 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:56.433173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" event={"ID":"d3befa73-ddfc-4363-b5dc-450fe17bf51a","Type":"ContainerStarted","Data":"4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2"} Sep 11 15:25:56.433670 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:56.433241 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" event={"ID":"d3befa73-ddfc-4363-b5dc-450fe17bf51a","Type":"ContainerStarted","Data":"59a9bbe6e9026522bb51688518b317b753324c8bbaa3273c71e4dca95126af49"} Sep 11 15:25:56.433670 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:56.433364 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:25:56.435100 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:56.435076 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" event={"ID":"7eda975c-26f7-415c-9efe-508b99368610","Type":"ContainerStarted","Data":"ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6"} Sep 11 15:25:56.435206 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:56.435122 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" event={"ID":"7eda975c-26f7-415c-9efe-508b99368610","Type":"ContainerStarted","Data":"1316b1a24578ef3b3b81e45445ae0a594d321e0f59bf5eaf99cc53ef5f4fce5b"} Sep 11 15:25:56.435206 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:56.435149 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:25:56.453409 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:56.453372 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" podStartSLOduration=1.453359255 podStartE2EDuration="1.453359255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:25:56.450649263 +0000 UTC m=+1038.495974245" watchObservedRunningTime="2026-09-11 15:25:56.453359255 +0000 UTC m=+1038.498684236" Sep 11 15:25:56.466610 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:25:56.466551 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" podStartSLOduration=1.4665365719999999 podStartE2EDuration="1.466536572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:25:56.46631988 +0000 UTC m=+1038.511644862" watchObservedRunningTime="2026-09-11 15:25:56.466536572 +0000 UTC m=+1038.511861545" Sep 11 15:26:07.441681 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:07.441650 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:26:07.442146 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:07.441719 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:26:12.183672 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.183631 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk"] Sep 11 15:26:12.187598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.187574 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.190295 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.190273 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d3b1ea\"/\"openshift-service-ca.crt\"" Sep 11 15:26:12.190447 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.190426 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d3b1ea\"/\"kube-root-ca.crt\"" Sep 11 15:26:12.191349 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.191327 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d3b1ea\"/\"default-dockercfg-zh85f\"" Sep 11 15:26:12.191457 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.191372 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d3b1ea\"/\"noten-model-d3b1ea-kserve-self-signed-certs\"" Sep 11 15:26:12.196367 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.196346 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk"] Sep 11 15:26:12.355882 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.355848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxgr\" (UniqueName: \"kubernetes.io/projected/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-kube-api-access-7zxgr\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.356040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.355914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tmp-dir\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.356040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.355980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tls-certs\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.356171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.356053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-dshm\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.356240 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.356175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-model-cache\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.356240 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.356223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-home\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.456671 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.456608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-model-cache\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.456671 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.456639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-home\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.456855 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.456800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxgr\" (UniqueName: \"kubernetes.io/projected/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-kube-api-access-7zxgr\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.456926 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.456885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tmp-dir\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.456983 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.456934 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tls-certs\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.457040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.456987 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-home\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.457040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.457001 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-model-cache\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.457040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.456989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-dshm\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.457205 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.457187 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tmp-dir\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.459244 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.459224 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-dshm\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.459534 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.459513 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tls-certs\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.465419 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.465398 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxgr\" (UniqueName: \"kubernetes.io/projected/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-kube-api-access-7zxgr\") pod \"noten-model-d3b1ea-kserve-6465c8657f-qf8nk\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.511495 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.511471 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:12.781913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:12.781884 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk"] Sep 11 15:26:12.783634 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:26:12.783600 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b1e8b33_cb47_4c52_b8e5_0ebfcbf9462d.slice/crio-285e45472479ac0b045da4ee23c4f27fbbb4fd1a287e44e7147779ca53ff49f8 WatchSource:0}: Error finding container 285e45472479ac0b045da4ee23c4f27fbbb4fd1a287e44e7147779ca53ff49f8: Status 404 returned error can't find the container with id 285e45472479ac0b045da4ee23c4f27fbbb4fd1a287e44e7147779ca53ff49f8 Sep 11 15:26:13.525635 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:13.525603 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" event={"ID":"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d","Type":"ContainerStarted","Data":"6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab"} Sep 11 15:26:13.525635 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:13.525638 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" event={"ID":"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d","Type":"ContainerStarted","Data":"285e45472479ac0b045da4ee23c4f27fbbb4fd1a287e44e7147779ca53ff49f8"} Sep 11 15:26:13.546260 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:13.543252 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" podStartSLOduration=1.543224811 podStartE2EDuration="1.543224811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:26:13.540934245 +0000 UTC m=+1055.586259224" watchObservedRunningTime="2026-09-11 15:26:13.543224811 +0000 UTC m=+1055.588549791" Sep 11 15:26:15.771453 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:15.771421 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk"] Sep 11 15:26:17.130134 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:17.130091 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8"] Sep 11 15:26:17.133724 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:17.133706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8" Sep 11 15:26:17.136018 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:17.135987 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:26:17.136202 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:17.136183 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:26:17.141679 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:17.141472 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8"] Sep 11 15:26:17.295575 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:17.295541 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c262b6-330a-45f0-9bab-b8bef1a3817b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-778341-hdlg8\" (UID: \"86c262b6-330a-45f0-9bab-b8bef1a3817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8" Sep 11 15:26:17.397085 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:17.397006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c262b6-330a-45f0-9bab-b8bef1a3817b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-778341-hdlg8\" (UID: \"86c262b6-330a-45f0-9bab-b8bef1a3817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8" Sep 11 15:26:17.397588 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:17.397568 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c262b6-330a-45f0-9bab-b8bef1a3817b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-778341-hdlg8\" (UID: \"86c262b6-330a-45f0-9bab-b8bef1a3817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8" Sep 11 15:26:17.445052 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:17.445029 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8" Sep 11 15:26:17.716867 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:17.716839 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8"] Sep 11 15:26:17.719052 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:26:17.719024 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c262b6_330a_45f0_9bab_b8bef1a3817b.slice/crio-c4615739414458444339ee4ef8d7f50cff3951eae41a14542f3fa9d8c05cb851 WatchSource:0}: Error finding container c4615739414458444339ee4ef8d7f50cff3951eae41a14542f3fa9d8c05cb851: Status 404 returned error can't find the container with id c4615739414458444339ee4ef8d7f50cff3951eae41a14542f3fa9d8c05cb851 Sep 11 15:26:18.552907 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:18.552815 2573 generic.go:356] "Generic (PLEG): container finished" podID="86c262b6-330a-45f0-9bab-b8bef1a3817b" containerID="d4c032081d021d7631f9e234c6576e0244c0e66b81d79af4f9dc6372675aaf95" exitCode=0 Sep 11 15:26:18.552907 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:18.552880 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8" event={"ID":"86c262b6-330a-45f0-9bab-b8bef1a3817b","Type":"ContainerDied","Data":"d4c032081d021d7631f9e234c6576e0244c0e66b81d79af4f9dc6372675aaf95"} Sep 11 15:26:18.553311 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:18.552919 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8" event={"ID":"86c262b6-330a-45f0-9bab-b8bef1a3817b","Type":"ContainerStarted","Data":"c4615739414458444339ee4ef8d7f50cff3951eae41a14542f3fa9d8c05cb851"} Sep 11 15:26:19.688389 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:19.688362 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8" Sep 11 15:26:19.816336 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:19.816264 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c262b6-330a-45f0-9bab-b8bef1a3817b-maas-api-service-ca\") pod \"86c262b6-330a-45f0-9bab-b8bef1a3817b\" (UID: \"86c262b6-330a-45f0-9bab-b8bef1a3817b\") " Sep 11 15:26:19.816583 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:19.816559 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c262b6-330a-45f0-9bab-b8bef1a3817b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86c262b6-330a-45f0-9bab-b8bef1a3817b" (UID: "86c262b6-330a-45f0-9bab-b8bef1a3817b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:19.918048 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:19.918020 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c262b6-330a-45f0-9bab-b8bef1a3817b-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:20.564381 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:20.564348 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8" event={"ID":"86c262b6-330a-45f0-9bab-b8bef1a3817b","Type":"ContainerDied","Data":"c4615739414458444339ee4ef8d7f50cff3951eae41a14542f3fa9d8c05cb851"} Sep 11 15:26:20.564529 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:20.564393 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4615739414458444339ee4ef8d7f50cff3951eae41a14542f3fa9d8c05cb851" Sep 11 15:26:20.564529 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:20.564356 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8" Sep 11 15:26:22.511949 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:22.511912 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:25.018982 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.018946 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh"] Sep 11 15:26:25.019355 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.019342 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c262b6-330a-45f0-9bab-b8bef1a3817b" containerName="revoke-keys" Sep 11 15:26:25.019355 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.019355 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c262b6-330a-45f0-9bab-b8bef1a3817b" containerName="revoke-keys" Sep 11 15:26:25.019431 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.019417 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c262b6-330a-45f0-9bab-b8bef1a3817b" containerName="revoke-keys" Sep 11 15:26:25.022552 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.022537 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh" Sep 11 15:26:25.024742 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.024713 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:26:25.024884 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.024761 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:26:25.029938 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.029718 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh"] Sep 11 15:26:25.055969 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.055940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b453515-d638-4229-87c2-fa1150a67877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0dd91403-msfxh\" (UID: \"5b453515-d638-4229-87c2-fa1150a67877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh" Sep 11 15:26:25.157224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.157197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b453515-d638-4229-87c2-fa1150a67877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0dd91403-msfxh\" (UID: \"5b453515-d638-4229-87c2-fa1150a67877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh" Sep 11 15:26:25.157767 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.157750 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b453515-d638-4229-87c2-fa1150a67877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0dd91403-msfxh\" (UID: \"5b453515-d638-4229-87c2-fa1150a67877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh" Sep 11 15:26:25.334860 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.334838 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh" Sep 11 15:26:25.603959 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:25.603936 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh"] Sep 11 15:26:25.606049 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:26:25.606015 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b453515_d638_4229_87c2_fa1150a67877.slice/crio-58a20088702fbcc7a9141cf2b0e066318b5633c7df1370c06c972aaee673decd WatchSource:0}: Error finding container 58a20088702fbcc7a9141cf2b0e066318b5633c7df1370c06c972aaee673decd: Status 404 returned error can't find the container with id 58a20088702fbcc7a9141cf2b0e066318b5633c7df1370c06c972aaee673decd Sep 11 15:26:26.596147 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:26.596119 2573 generic.go:356] "Generic (PLEG): container finished" podID="5b453515-d638-4229-87c2-fa1150a67877" containerID="b91312ce4f722735411a3c58ed8a90088b8b32eb529ba197668904eac2c7d1a5" exitCode=0 Sep 11 15:26:26.596583 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:26.596205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh" event={"ID":"5b453515-d638-4229-87c2-fa1150a67877","Type":"ContainerDied","Data":"b91312ce4f722735411a3c58ed8a90088b8b32eb529ba197668904eac2c7d1a5"} Sep 11 15:26:26.596583 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:26.596264 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh" event={"ID":"5b453515-d638-4229-87c2-fa1150a67877","Type":"ContainerStarted","Data":"58a20088702fbcc7a9141cf2b0e066318b5633c7df1370c06c972aaee673decd"} Sep 11 15:26:27.736468 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:27.736440 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh" Sep 11 15:26:27.778534 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:27.778502 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b453515-d638-4229-87c2-fa1150a67877-maas-api-service-ca\") pod \"5b453515-d638-4229-87c2-fa1150a67877\" (UID: \"5b453515-d638-4229-87c2-fa1150a67877\") " Sep 11 15:26:27.778955 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:27.778918 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b453515-d638-4229-87c2-fa1150a67877-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b453515-d638-4229-87c2-fa1150a67877" (UID: "5b453515-d638-4229-87c2-fa1150a67877"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:27.879455 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:27.879387 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b453515-d638-4229-87c2-fa1150a67877-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:28.608270 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:28.608240 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh" Sep 11 15:26:28.608400 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:28.608239 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh" event={"ID":"5b453515-d638-4229-87c2-fa1150a67877","Type":"ContainerDied","Data":"58a20088702fbcc7a9141cf2b0e066318b5633c7df1370c06c972aaee673decd"} Sep 11 15:26:28.608400 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:28.608360 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a20088702fbcc7a9141cf2b0e066318b5633c7df1370c06c972aaee673decd" Sep 11 15:26:28.984430 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:28.984398 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc"] Sep 11 15:26:28.984905 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:28.984653 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" podUID="17e643a4-8ab8-4621-ace9-3bbc56c0fab8" containerName="payload-processing" containerID="cri-o://c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277" gracePeriod=30 Sep 11 15:26:28.991834 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:28.991807 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7"] Sep 11 15:26:28.992009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:28.991975 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" podUID="42c42ed8-71fe-492e-80c6-97a5b2c864ad" containerName="payload-pre-processing" containerID="cri-o://afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44" gracePeriod=30 Sep 11 15:26:29.014672 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.014554 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" podUID="17e643a4-8ab8-4621-ace9-3bbc56c0fab8" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.38:9004: connect: connection refused" Sep 11 15:26:29.149697 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.149655 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8"] Sep 11 15:26:29.152679 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.152601 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-778341-hdlg8"] Sep 11 15:26:29.244991 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.244932 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:26:29.272556 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.272529 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:26:29.293016 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.292985 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrc5w\" (UniqueName: \"kubernetes.io/projected/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-kube-api-access-jrc5w\") pod \"17e643a4-8ab8-4621-ace9-3bbc56c0fab8\" (UID: \"17e643a4-8ab8-4621-ace9-3bbc56c0fab8\") " Sep 11 15:26:29.293139 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.293099 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-plugins-config-volume\") pod \"17e643a4-8ab8-4621-ace9-3bbc56c0fab8\" (UID: \"17e643a4-8ab8-4621-ace9-3bbc56c0fab8\") " Sep 11 15:26:29.293139 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.293136 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c42ed8-71fe-492e-80c6-97a5b2c864ad-plugins-config-volume\") pod \"42c42ed8-71fe-492e-80c6-97a5b2c864ad\" (UID: \"42c42ed8-71fe-492e-80c6-97a5b2c864ad\") " Sep 11 15:26:29.293261 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.293156 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l4x7\" (UniqueName: \"kubernetes.io/projected/42c42ed8-71fe-492e-80c6-97a5b2c864ad-kube-api-access-7l4x7\") pod \"42c42ed8-71fe-492e-80c6-97a5b2c864ad\" (UID: \"42c42ed8-71fe-492e-80c6-97a5b2c864ad\") " Sep 11 15:26:29.293567 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.293540 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c42ed8-71fe-492e-80c6-97a5b2c864ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42c42ed8-71fe-492e-80c6-97a5b2c864ad" (UID: "42c42ed8-71fe-492e-80c6-97a5b2c864ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:29.293879 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.293727 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17e643a4-8ab8-4621-ace9-3bbc56c0fab8" (UID: "17e643a4-8ab8-4621-ace9-3bbc56c0fab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:29.295479 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.295457 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-kube-api-access-jrc5w" (OuterVolumeSpecName: "kube-api-access-jrc5w") pod "17e643a4-8ab8-4621-ace9-3bbc56c0fab8" (UID: "17e643a4-8ab8-4621-ace9-3bbc56c0fab8"). InnerVolumeSpecName "kube-api-access-jrc5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:29.295582 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.295554 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c42ed8-71fe-492e-80c6-97a5b2c864ad-kube-api-access-7l4x7" (OuterVolumeSpecName: "kube-api-access-7l4x7") pod "42c42ed8-71fe-492e-80c6-97a5b2c864ad" (UID: "42c42ed8-71fe-492e-80c6-97a5b2c864ad"). InnerVolumeSpecName "kube-api-access-7l4x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:29.394119 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.394078 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrc5w\" (UniqueName: \"kubernetes.io/projected/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-kube-api-access-jrc5w\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:29.394119 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.394119 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e643a4-8ab8-4621-ace9-3bbc56c0fab8-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:29.394242 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.394130 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c42ed8-71fe-492e-80c6-97a5b2c864ad-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:29.394242 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.394139 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l4x7\" (UniqueName: \"kubernetes.io/projected/42c42ed8-71fe-492e-80c6-97a5b2c864ad-kube-api-access-7l4x7\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:29.615155 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.615100 2573 generic.go:356] "Generic (PLEG): container finished" podID="42c42ed8-71fe-492e-80c6-97a5b2c864ad" containerID="afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44" exitCode=0 Sep 11 15:26:29.615155 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.615146 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" Sep 11 15:26:29.615334 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.615179 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" event={"ID":"42c42ed8-71fe-492e-80c6-97a5b2c864ad","Type":"ContainerDied","Data":"afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44"} Sep 11 15:26:29.615334 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.615237 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7" event={"ID":"42c42ed8-71fe-492e-80c6-97a5b2c864ad","Type":"ContainerDied","Data":"b7c1103f47c019635b6e95275cd8156b2bebea63f649288fabbba03e2dc5a3ff"} Sep 11 15:26:29.615334 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.615269 2573 scope.go:117] "RemoveContainer" containerID="afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44" Sep 11 15:26:29.619793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.616930 2573 generic.go:356] "Generic (PLEG): container finished" podID="17e643a4-8ab8-4621-ace9-3bbc56c0fab8" containerID="c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277" exitCode=0 Sep 11 15:26:29.619793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.617009 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" event={"ID":"17e643a4-8ab8-4621-ace9-3bbc56c0fab8","Type":"ContainerDied","Data":"c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277"} Sep 11 15:26:29.619793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.617051 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" event={"ID":"17e643a4-8ab8-4621-ace9-3bbc56c0fab8","Type":"ContainerDied","Data":"f9e0303d6b328eea2f415e3f072eb1aaf37a136c6bbf3025eae2675c0be7f2d3"} Sep 11 15:26:29.619793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.617114 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc" Sep 11 15:26:29.628081 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.628064 2573 scope.go:117] "RemoveContainer" containerID="afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44" Sep 11 15:26:29.628592 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:26:29.628369 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44\": container with ID starting with afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44 not found: ID does not exist" containerID="afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44" Sep 11 15:26:29.628592 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.628403 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44"} err="failed to get container status \"afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44\": rpc error: code = NotFound desc = could not find container \"afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44\": container with ID starting with afffea76681d82cf2484a96af7231e32e3c73f7993c0aa6cf42d7875a5d28b44 not found: ID does not exist" Sep 11 15:26:29.628592 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.628426 2573 scope.go:117] "RemoveContainer" containerID="c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277" Sep 11 15:26:29.636419 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.636390 2573 scope.go:117] "RemoveContainer" containerID="c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277" Sep 11 15:26:29.636689 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:26:29.636672 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277\": container with ID starting with c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277 not found: ID does not exist" containerID="c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277" Sep 11 15:26:29.636736 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.636696 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277"} err="failed to get container status \"c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277\": rpc error: code = NotFound desc = could not find container \"c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277\": container with ID starting with c793eb6b60df0b7e39357a9c489b058eaf197cfa6f23028725156ff1c29ad277 not found: ID does not exist" Sep 11 15:26:29.639807 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.639788 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7"] Sep 11 15:26:29.645995 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.645976 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-778341-d5f4fcf5d-vvcz7"] Sep 11 15:26:29.656162 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.656142 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc"] Sep 11 15:26:29.660106 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:29.660087 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-778341-c59cb47b5-fm2gc"] Sep 11 15:26:30.493685 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:30.493651 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e643a4-8ab8-4621-ace9-3bbc56c0fab8" path="/var/lib/kubelet/pods/17e643a4-8ab8-4621-ace9-3bbc56c0fab8/volumes" Sep 11 15:26:30.494056 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:30.494031 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c42ed8-71fe-492e-80c6-97a5b2c864ad" path="/var/lib/kubelet/pods/42c42ed8-71fe-492e-80c6-97a5b2c864ad/volumes" Sep 11 15:26:30.494356 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:30.494341 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c262b6-330a-45f0-9bab-b8bef1a3817b" path="/var/lib/kubelet/pods/86c262b6-330a-45f0-9bab-b8bef1a3817b/volumes" Sep 11 15:26:30.791983 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:30.791904 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" podUID="6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" containerName="main" containerID="cri-o://6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab" gracePeriod=45 Sep 11 15:26:31.047567 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.047512 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:31.108488 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.108464 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tmp-dir\") pod \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " Sep 11 15:26:31.108635 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.108523 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tls-certs\") pod \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " Sep 11 15:26:31.108635 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.108565 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-home\") pod \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " Sep 11 15:26:31.108635 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.108608 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-dshm\") pod \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " Sep 11 15:26:31.108842 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.108641 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zxgr\" (UniqueName: \"kubernetes.io/projected/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-kube-api-access-7zxgr\") pod \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " Sep 11 15:26:31.108842 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.108674 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-model-cache\") pod \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\" (UID: \"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d\") " Sep 11 15:26:31.108842 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.108726 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" (UID: "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:31.109007 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.108824 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-home" (OuterVolumeSpecName: "home") pod "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" (UID: "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:31.109007 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.108965 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" (UID: "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:31.109202 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.109183 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tmp-dir\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:31.109251 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.109209 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-home\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:31.109251 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.109224 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:31.110968 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.110944 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-dshm" (OuterVolumeSpecName: "dshm") pod "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" (UID: "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:31.111066 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.110963 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-kube-api-access-7zxgr" (OuterVolumeSpecName: "kube-api-access-7zxgr") pod "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" (UID: "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d"). InnerVolumeSpecName "kube-api-access-7zxgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:31.111066 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.110944 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" (UID: "6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:26:31.210228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.210204 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-tls-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:31.210228 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.210227 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-dshm\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:31.210376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.210236 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zxgr\" (UniqueName: \"kubernetes.io/projected/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d-kube-api-access-7zxgr\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:31.630711 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.630681 2573 generic.go:356] "Generic (PLEG): container finished" podID="6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" containerID="6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab" exitCode=0 Sep 11 15:26:31.631114 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.630730 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" Sep 11 15:26:31.631114 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.630751 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" event={"ID":"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d","Type":"ContainerDied","Data":"6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab"} Sep 11 15:26:31.631114 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.630800 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk" event={"ID":"6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d","Type":"ContainerDied","Data":"285e45472479ac0b045da4ee23c4f27fbbb4fd1a287e44e7147779ca53ff49f8"} Sep 11 15:26:31.631114 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.630824 2573 scope.go:117] "RemoveContainer" containerID="6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab" Sep 11 15:26:31.640628 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.640612 2573 scope.go:117] "RemoveContainer" containerID="6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab" Sep 11 15:26:31.640954 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:26:31.640932 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab\": container with ID starting with 6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab not found: ID does not exist" containerID="6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab" Sep 11 15:26:31.641044 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.640959 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab"} err="failed to get container status \"6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab\": rpc error: code = NotFound desc = could not find container \"6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab\": container with ID starting with 6496dc88e6acbb3c4c5421c9854c0b2f010dda929b51f12efdf86363b1faf7ab not found: ID does not exist" Sep 11 15:26:31.653697 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.653672 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk"] Sep 11 15:26:31.658220 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:31.658196 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-d3b1ea/noten-model-d3b1ea-kserve-6465c8657f-qf8nk"] Sep 11 15:26:32.493290 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:32.493242 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" path="/var/lib/kubelet/pods/6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d/volumes" Sep 11 15:26:35.749890 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:35.749860 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44"] Sep 11 15:26:35.750344 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:35.750104 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" podUID="d3befa73-ddfc-4363-b5dc-450fe17bf51a" containerName="payload-processing" containerID="cri-o://4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2" gracePeriod=30 Sep 11 15:26:35.758277 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:35.758248 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt"] Sep 11 15:26:35.758549 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:35.758463 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" podUID="7eda975c-26f7-415c-9efe-508b99368610" containerName="payload-pre-processing" containerID="cri-o://ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6" gracePeriod=30 Sep 11 15:26:35.916970 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:35.916941 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh"] Sep 11 15:26:35.920170 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:35.920119 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0dd91403-msfxh"] Sep 11 15:26:35.999499 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:35.999474 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:26:36.037735 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.037709 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:26:36.048759 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.048733 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3befa73-ddfc-4363-b5dc-450fe17bf51a-plugins-config-volume\") pod \"d3befa73-ddfc-4363-b5dc-450fe17bf51a\" (UID: \"d3befa73-ddfc-4363-b5dc-450fe17bf51a\") " Sep 11 15:26:36.048878 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.048831 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5b62\" (UniqueName: \"kubernetes.io/projected/d3befa73-ddfc-4363-b5dc-450fe17bf51a-kube-api-access-g5b62\") pod \"d3befa73-ddfc-4363-b5dc-450fe17bf51a\" (UID: \"d3befa73-ddfc-4363-b5dc-450fe17bf51a\") " Sep 11 15:26:36.049179 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.049153 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3befa73-ddfc-4363-b5dc-450fe17bf51a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3befa73-ddfc-4363-b5dc-450fe17bf51a" (UID: "d3befa73-ddfc-4363-b5dc-450fe17bf51a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:36.050926 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.050908 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3befa73-ddfc-4363-b5dc-450fe17bf51a-kube-api-access-g5b62" (OuterVolumeSpecName: "kube-api-access-g5b62") pod "d3befa73-ddfc-4363-b5dc-450fe17bf51a" (UID: "d3befa73-ddfc-4363-b5dc-450fe17bf51a"). InnerVolumeSpecName "kube-api-access-g5b62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:36.149903 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.149881 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eda975c-26f7-415c-9efe-508b99368610-plugins-config-volume\") pod \"7eda975c-26f7-415c-9efe-508b99368610\" (UID: \"7eda975c-26f7-415c-9efe-508b99368610\") " Sep 11 15:26:36.150030 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.149969 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhmld\" (UniqueName: \"kubernetes.io/projected/7eda975c-26f7-415c-9efe-508b99368610-kube-api-access-rhmld\") pod \"7eda975c-26f7-415c-9efe-508b99368610\" (UID: \"7eda975c-26f7-415c-9efe-508b99368610\") " Sep 11 15:26:36.150184 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.150163 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5b62\" (UniqueName: \"kubernetes.io/projected/d3befa73-ddfc-4363-b5dc-450fe17bf51a-kube-api-access-g5b62\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:36.150184 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.150180 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3befa73-ddfc-4363-b5dc-450fe17bf51a-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:36.150318 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.150237 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eda975c-26f7-415c-9efe-508b99368610-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7eda975c-26f7-415c-9efe-508b99368610" (UID: "7eda975c-26f7-415c-9efe-508b99368610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:36.152077 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.152056 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eda975c-26f7-415c-9efe-508b99368610-kube-api-access-rhmld" (OuterVolumeSpecName: "kube-api-access-rhmld") pod "7eda975c-26f7-415c-9efe-508b99368610" (UID: "7eda975c-26f7-415c-9efe-508b99368610"). InnerVolumeSpecName "kube-api-access-rhmld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:36.250935 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.250914 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eda975c-26f7-415c-9efe-508b99368610-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:36.250935 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.250934 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhmld\" (UniqueName: \"kubernetes.io/projected/7eda975c-26f7-415c-9efe-508b99368610-kube-api-access-rhmld\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:36.492833 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.492805 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b453515-d638-4229-87c2-fa1150a67877" path="/var/lib/kubelet/pods/5b453515-d638-4229-87c2-fa1150a67877/volumes" Sep 11 15:26:36.657023 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.656992 2573 generic.go:356] "Generic (PLEG): container finished" podID="7eda975c-26f7-415c-9efe-508b99368610" containerID="ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6" exitCode=0 Sep 11 15:26:36.657133 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.657038 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" Sep 11 15:26:36.657133 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.657071 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" event={"ID":"7eda975c-26f7-415c-9efe-508b99368610","Type":"ContainerDied","Data":"ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6"} Sep 11 15:26:36.657133 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.657120 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt" event={"ID":"7eda975c-26f7-415c-9efe-508b99368610","Type":"ContainerDied","Data":"1316b1a24578ef3b3b81e45445ae0a594d321e0f59bf5eaf99cc53ef5f4fce5b"} Sep 11 15:26:36.657241 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.657141 2573 scope.go:117] "RemoveContainer" containerID="ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6" Sep 11 15:26:36.659189 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.659154 2573 generic.go:356] "Generic (PLEG): container finished" podID="d3befa73-ddfc-4363-b5dc-450fe17bf51a" containerID="4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2" exitCode=0 Sep 11 15:26:36.659346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.659326 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" event={"ID":"d3befa73-ddfc-4363-b5dc-450fe17bf51a","Type":"ContainerDied","Data":"4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2"} Sep 11 15:26:36.659439 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.659379 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" event={"ID":"d3befa73-ddfc-4363-b5dc-450fe17bf51a","Type":"ContainerDied","Data":"59a9bbe6e9026522bb51688518b317b753324c8bbaa3273c71e4dca95126af49"} Sep 11 15:26:36.659439 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.659401 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44" Sep 11 15:26:36.672006 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.671989 2573 scope.go:117] "RemoveContainer" containerID="ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6" Sep 11 15:26:36.672389 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:26:36.672354 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6\": container with ID starting with ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6 not found: ID does not exist" containerID="ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6" Sep 11 15:26:36.672466 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.672405 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6"} err="failed to get container status \"ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6\": rpc error: code = NotFound desc = could not find container \"ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6\": container with ID starting with ad5b34aa976cba0fda56c3b0a7dd105189d6a9483b6aedf0bc67d7422cdf81a6 not found: ID does not exist" Sep 11 15:26:36.672466 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.672433 2573 scope.go:117] "RemoveContainer" containerID="4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2" Sep 11 15:26:36.678199 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.678176 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt"] Sep 11 15:26:36.681118 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.681100 2573 scope.go:117] "RemoveContainer" containerID="4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2" Sep 11 15:26:36.681436 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:26:36.681419 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2\": container with ID starting with 4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2 not found: ID does not exist" containerID="4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2" Sep 11 15:26:36.681484 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.681444 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2"} err="failed to get container status \"4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2\": rpc error: code = NotFound desc = could not find container \"4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2\": container with ID starting with 4af3cdd2316a980ce2740cf70175fe8a9770defd970c89ca831deaec8a6b02b2 not found: ID does not exist" Sep 11 15:26:36.681754 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.681739 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0dd91403-56f6bcbb5d-grbmt"] Sep 11 15:26:36.695545 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.695525 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44"] Sep 11 15:26:36.699452 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.699432 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0dd91403-788f5d99c7-zwg44"] Sep 11 15:26:36.792177 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792152 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg"] Sep 11 15:26:36.792675 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792656 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3befa73-ddfc-4363-b5dc-450fe17bf51a" containerName="payload-processing" Sep 11 15:26:36.792751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792678 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3befa73-ddfc-4363-b5dc-450fe17bf51a" containerName="payload-processing" Sep 11 15:26:36.792751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792698 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eda975c-26f7-415c-9efe-508b99368610" containerName="payload-pre-processing" Sep 11 15:26:36.792751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792707 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eda975c-26f7-415c-9efe-508b99368610" containerName="payload-pre-processing" Sep 11 15:26:36.792751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792718 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c42ed8-71fe-492e-80c6-97a5b2c864ad" containerName="payload-pre-processing" Sep 11 15:26:36.792751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792728 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c42ed8-71fe-492e-80c6-97a5b2c864ad" containerName="payload-pre-processing" Sep 11 15:26:36.792751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792741 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e643a4-8ab8-4621-ace9-3bbc56c0fab8" containerName="payload-processing" Sep 11 15:26:36.792751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792749 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e643a4-8ab8-4621-ace9-3bbc56c0fab8" containerName="payload-processing" Sep 11 15:26:36.793124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792764 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" containerName="main" Sep 11 15:26:36.793124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792790 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" containerName="main" Sep 11 15:26:36.793124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792803 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b453515-d638-4229-87c2-fa1150a67877" containerName="revoke-keys" Sep 11 15:26:36.793124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792813 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b453515-d638-4229-87c2-fa1150a67877" containerName="revoke-keys" Sep 11 15:26:36.793124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792917 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3befa73-ddfc-4363-b5dc-450fe17bf51a" containerName="payload-processing" Sep 11 15:26:36.793124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792930 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c42ed8-71fe-492e-80c6-97a5b2c864ad" containerName="payload-pre-processing" Sep 11 15:26:36.793124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792940 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b1e8b33-cb47-4c52-b8e5-0ebfcbf9462d" containerName="main" Sep 11 15:26:36.793124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792950 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eda975c-26f7-415c-9efe-508b99368610" containerName="payload-pre-processing" Sep 11 15:26:36.793124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792959 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b453515-d638-4229-87c2-fa1150a67877" containerName="revoke-keys" Sep 11 15:26:36.793124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.792969 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e643a4-8ab8-4621-ace9-3bbc56c0fab8" containerName="payload-processing" Sep 11 15:26:36.797867 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.797850 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg" Sep 11 15:26:36.799977 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.799953 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:26:36.800066 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.800002 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:26:36.802828 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.802807 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg"] Sep 11 15:26:36.856058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.856030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb2ed94-c4a3-4252-9a4e-9f6e35119044-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg\" (UID: \"beb2ed94-c4a3-4252-9a4e-9f6e35119044\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg" Sep 11 15:26:36.957029 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.957006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb2ed94-c4a3-4252-9a4e-9f6e35119044-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg\" (UID: \"beb2ed94-c4a3-4252-9a4e-9f6e35119044\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg" Sep 11 15:26:36.957597 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:36.957581 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb2ed94-c4a3-4252-9a4e-9f6e35119044-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg\" (UID: \"beb2ed94-c4a3-4252-9a4e-9f6e35119044\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg" Sep 11 15:26:37.108887 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:37.108864 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg" Sep 11 15:26:37.378800 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:37.378749 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg"] Sep 11 15:26:37.380576 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:26:37.380545 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb2ed94_c4a3_4252_9a4e_9f6e35119044.slice/crio-0836f4fcc001318764a89743cbab1ecae35ec5d4cb7a5bf35d1282eb2c6629d3 WatchSource:0}: Error finding container 0836f4fcc001318764a89743cbab1ecae35ec5d4cb7a5bf35d1282eb2c6629d3: Status 404 returned error can't find the container with id 0836f4fcc001318764a89743cbab1ecae35ec5d4cb7a5bf35d1282eb2c6629d3 Sep 11 15:26:37.667914 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:37.667849 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg" event={"ID":"beb2ed94-c4a3-4252-9a4e-9f6e35119044","Type":"ContainerStarted","Data":"0836f4fcc001318764a89743cbab1ecae35ec5d4cb7a5bf35d1282eb2c6629d3"} Sep 11 15:26:38.493498 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:38.493423 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eda975c-26f7-415c-9efe-508b99368610" path="/var/lib/kubelet/pods/7eda975c-26f7-415c-9efe-508b99368610/volumes" Sep 11 15:26:38.493843 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:38.493794 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3befa73-ddfc-4363-b5dc-450fe17bf51a" path="/var/lib/kubelet/pods/d3befa73-ddfc-4363-b5dc-450fe17bf51a/volumes" Sep 11 15:26:38.675265 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:38.675237 2573 generic.go:356] "Generic (PLEG): container finished" podID="beb2ed94-c4a3-4252-9a4e-9f6e35119044" containerID="96a208c9bb9ce9bf1f288296324dd77196672d5d955e6cd5ac12d6ff7a0d22ef" exitCode=0 Sep 11 15:26:38.675385 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:38.675318 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg" event={"ID":"beb2ed94-c4a3-4252-9a4e-9f6e35119044","Type":"ContainerDied","Data":"96a208c9bb9ce9bf1f288296324dd77196672d5d955e6cd5ac12d6ff7a0d22ef"} Sep 11 15:26:39.813169 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:39.813145 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg" Sep 11 15:26:39.882485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:39.882457 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb2ed94-c4a3-4252-9a4e-9f6e35119044-maas-api-service-ca\") pod \"beb2ed94-c4a3-4252-9a4e-9f6e35119044\" (UID: \"beb2ed94-c4a3-4252-9a4e-9f6e35119044\") " Sep 11 15:26:39.882760 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:39.882738 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb2ed94-c4a3-4252-9a4e-9f6e35119044-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "beb2ed94-c4a3-4252-9a4e-9f6e35119044" (UID: "beb2ed94-c4a3-4252-9a4e-9f6e35119044"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:39.984119 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:39.984059 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb2ed94-c4a3-4252-9a4e-9f6e35119044-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:40.686216 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:40.686191 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg" Sep 11 15:26:40.686216 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:40.686205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg" event={"ID":"beb2ed94-c4a3-4252-9a4e-9f6e35119044","Type":"ContainerDied","Data":"0836f4fcc001318764a89743cbab1ecae35ec5d4cb7a5bf35d1282eb2c6629d3"} Sep 11 15:26:40.686412 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:40.686250 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0836f4fcc001318764a89743cbab1ecae35ec5d4cb7a5bf35d1282eb2c6629d3" Sep 11 15:26:47.564649 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.564613 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg"] Sep 11 15:26:47.565132 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.564920 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" podUID="1bc6eed3-1e2c-48af-87b9-7833e2206495" containerName="payload-processing" containerID="cri-o://041579c33527729dd436625636477d5a2d2351a8a2b0fc0fd9a17b97322da64c" gracePeriod=30 Sep 11 15:26:47.569874 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.569849 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f"] Sep 11 15:26:47.570196 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.570171 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" podUID="bfb087d5-d6f0-43d3-b38f-44900016a9e9" containerName="payload-pre-processing" containerID="cri-o://198eb0b2c3c97ca9b20c03b542ecb8bf0cb2c092ce1b689131b34fcf861da3cc" gracePeriod=30 Sep 11 15:26:47.712960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.712896 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg"] Sep 11 15:26:47.716703 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.716676 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f08bd-8nxfg"] Sep 11 15:26:47.723336 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.723310 2573 generic.go:356] "Generic (PLEG): container finished" podID="1bc6eed3-1e2c-48af-87b9-7833e2206495" containerID="041579c33527729dd436625636477d5a2d2351a8a2b0fc0fd9a17b97322da64c" exitCode=0 Sep 11 15:26:47.723461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.723366 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" event={"ID":"1bc6eed3-1e2c-48af-87b9-7833e2206495","Type":"ContainerDied","Data":"041579c33527729dd436625636477d5a2d2351a8a2b0fc0fd9a17b97322da64c"} Sep 11 15:26:47.725100 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.725075 2573 generic.go:356] "Generic (PLEG): container finished" podID="bfb087d5-d6f0-43d3-b38f-44900016a9e9" containerID="198eb0b2c3c97ca9b20c03b542ecb8bf0cb2c092ce1b689131b34fcf861da3cc" exitCode=0 Sep 11 15:26:47.725216 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.725114 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" event={"ID":"bfb087d5-d6f0-43d3-b38f-44900016a9e9","Type":"ContainerDied","Data":"198eb0b2c3c97ca9b20c03b542ecb8bf0cb2c092ce1b689131b34fcf861da3cc"} Sep 11 15:26:47.824309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.824291 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:26:47.847652 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.847626 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:26:47.946284 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.946257 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzqjs\" (UniqueName: \"kubernetes.io/projected/1bc6eed3-1e2c-48af-87b9-7833e2206495-kube-api-access-hzqjs\") pod \"1bc6eed3-1e2c-48af-87b9-7833e2206495\" (UID: \"1bc6eed3-1e2c-48af-87b9-7833e2206495\") " Sep 11 15:26:47.946398 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.946300 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb087d5-d6f0-43d3-b38f-44900016a9e9-plugins-config-volume\") pod \"bfb087d5-d6f0-43d3-b38f-44900016a9e9\" (UID: \"bfb087d5-d6f0-43d3-b38f-44900016a9e9\") " Sep 11 15:26:47.946461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.946429 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc6eed3-1e2c-48af-87b9-7833e2206495-plugins-config-volume\") pod \"1bc6eed3-1e2c-48af-87b9-7833e2206495\" (UID: \"1bc6eed3-1e2c-48af-87b9-7833e2206495\") " Sep 11 15:26:47.946508 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.946457 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjmkg\" (UniqueName: \"kubernetes.io/projected/bfb087d5-d6f0-43d3-b38f-44900016a9e9-kube-api-access-kjmkg\") pod \"bfb087d5-d6f0-43d3-b38f-44900016a9e9\" (UID: \"bfb087d5-d6f0-43d3-b38f-44900016a9e9\") " Sep 11 15:26:47.946825 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.946762 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb087d5-d6f0-43d3-b38f-44900016a9e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfb087d5-d6f0-43d3-b38f-44900016a9e9" (UID: "bfb087d5-d6f0-43d3-b38f-44900016a9e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:47.946825 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.946805 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc6eed3-1e2c-48af-87b9-7833e2206495-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1bc6eed3-1e2c-48af-87b9-7833e2206495" (UID: "1bc6eed3-1e2c-48af-87b9-7833e2206495"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:47.948515 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.948495 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc6eed3-1e2c-48af-87b9-7833e2206495-kube-api-access-hzqjs" (OuterVolumeSpecName: "kube-api-access-hzqjs") pod "1bc6eed3-1e2c-48af-87b9-7833e2206495" (UID: "1bc6eed3-1e2c-48af-87b9-7833e2206495"). InnerVolumeSpecName "kube-api-access-hzqjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:47.948579 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:47.948551 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb087d5-d6f0-43d3-b38f-44900016a9e9-kube-api-access-kjmkg" (OuterVolumeSpecName: "kube-api-access-kjmkg") pod "bfb087d5-d6f0-43d3-b38f-44900016a9e9" (UID: "bfb087d5-d6f0-43d3-b38f-44900016a9e9"). InnerVolumeSpecName "kube-api-access-kjmkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:48.048011 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.047986 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb087d5-d6f0-43d3-b38f-44900016a9e9-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:48.048011 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.048008 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc6eed3-1e2c-48af-87b9-7833e2206495-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:48.048138 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.048021 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjmkg\" (UniqueName: \"kubernetes.io/projected/bfb087d5-d6f0-43d3-b38f-44900016a9e9-kube-api-access-kjmkg\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:48.048138 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.048034 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzqjs\" (UniqueName: \"kubernetes.io/projected/1bc6eed3-1e2c-48af-87b9-7833e2206495-kube-api-access-hzqjs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:26:48.493447 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.493417 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beb2ed94-c4a3-4252-9a4e-9f6e35119044" path="/var/lib/kubelet/pods/beb2ed94-c4a3-4252-9a4e-9f6e35119044/volumes" Sep 11 15:26:48.732248 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.732223 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" Sep 11 15:26:48.732612 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.732219 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f" event={"ID":"bfb087d5-d6f0-43d3-b38f-44900016a9e9","Type":"ContainerDied","Data":"9502ca005460cb377d6a64f4bd64ed815d5f6a3a60f0d32c494fd05091566114"} Sep 11 15:26:48.732612 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.732385 2573 scope.go:117] "RemoveContainer" containerID="198eb0b2c3c97ca9b20c03b542ecb8bf0cb2c092ce1b689131b34fcf861da3cc" Sep 11 15:26:48.733614 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.733599 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" Sep 11 15:26:48.733672 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.733607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg" event={"ID":"1bc6eed3-1e2c-48af-87b9-7833e2206495","Type":"ContainerDied","Data":"46afcb989a3344a51dca80af9a81255580cced6cc1a6cfb7123bb9a31e4f905b"} Sep 11 15:26:48.741299 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.741277 2573 scope.go:117] "RemoveContainer" containerID="041579c33527729dd436625636477d5a2d2351a8a2b0fc0fd9a17b97322da64c" Sep 11 15:26:48.753222 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.753202 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f"] Sep 11 15:26:48.760268 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.760250 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f08bd-6b6cfdc69d-rxr4f"] Sep 11 15:26:48.775102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.775083 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg"] Sep 11 15:26:48.781408 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:48.781384 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f08bd-6d8945dcbf-bn6qg"] Sep 11 15:26:50.492598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:50.492562 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bc6eed3-1e2c-48af-87b9-7833e2206495" path="/var/lib/kubelet/pods/1bc6eed3-1e2c-48af-87b9-7833e2206495/volumes" Sep 11 15:26:50.492981 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:50.492940 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb087d5-d6f0-43d3-b38f-44900016a9e9" path="/var/lib/kubelet/pods/bfb087d5-d6f0-43d3-b38f-44900016a9e9/volumes" Sep 11 15:26:59.957530 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.957501 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx"] Sep 11 15:26:59.957997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.957878 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb2ed94-c4a3-4252-9a4e-9f6e35119044" containerName="revoke-keys" Sep 11 15:26:59.957997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.957889 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb2ed94-c4a3-4252-9a4e-9f6e35119044" containerName="revoke-keys" Sep 11 15:26:59.957997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.957902 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb087d5-d6f0-43d3-b38f-44900016a9e9" containerName="payload-pre-processing" Sep 11 15:26:59.957997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.957908 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb087d5-d6f0-43d3-b38f-44900016a9e9" containerName="payload-pre-processing" Sep 11 15:26:59.957997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.957931 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bc6eed3-1e2c-48af-87b9-7833e2206495" containerName="payload-processing" Sep 11 15:26:59.957997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.957937 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bc6eed3-1e2c-48af-87b9-7833e2206495" containerName="payload-processing" Sep 11 15:26:59.957997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.957991 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb087d5-d6f0-43d3-b38f-44900016a9e9" containerName="payload-pre-processing" Sep 11 15:26:59.957997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.958002 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb2ed94-c4a3-4252-9a4e-9f6e35119044" containerName="revoke-keys" Sep 11 15:26:59.958266 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.958010 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bc6eed3-1e2c-48af-87b9-7833e2206495" containerName="payload-processing" Sep 11 15:26:59.962898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.962870 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:26:59.964929 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.964907 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn"] Sep 11 15:26:59.965501 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.965480 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-397e40\"" Sep 11 15:26:59.965618 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.965600 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-397e40-dockercfg-2cp77\"" Sep 11 15:26:59.968306 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.968290 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:26:59.974059 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.974038 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx"] Sep 11 15:26:59.989229 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:26:59.989202 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn"] Sep 11 15:27:00.153085 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.153051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eadbec4-d689-4d16-b520-21110b97d007-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx\" (UID: \"6eadbec4-d689-4d16-b520-21110b97d007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:27:00.153215 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.153140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvm4\" (UniqueName: \"kubernetes.io/projected/6eadbec4-d689-4d16-b520-21110b97d007-kube-api-access-psvm4\") pod \"payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx\" (UID: \"6eadbec4-d689-4d16-b520-21110b97d007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:27:00.153215 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.153158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn\" (UID: \"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:27:00.153215 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.153177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n89k\" (UniqueName: \"kubernetes.io/projected/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-kube-api-access-7n89k\") pod \"payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn\" (UID: \"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:27:00.253820 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.253739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvm4\" (UniqueName: \"kubernetes.io/projected/6eadbec4-d689-4d16-b520-21110b97d007-kube-api-access-psvm4\") pod \"payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx\" (UID: \"6eadbec4-d689-4d16-b520-21110b97d007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:27:00.253820 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.253790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn\" (UID: \"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:27:00.253953 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.253901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n89k\" (UniqueName: \"kubernetes.io/projected/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-kube-api-access-7n89k\") pod \"payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn\" (UID: \"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:27:00.253997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.253947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eadbec4-d689-4d16-b520-21110b97d007-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx\" (UID: \"6eadbec4-d689-4d16-b520-21110b97d007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:27:00.254535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.254512 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn\" (UID: \"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:27:00.254663 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.254641 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eadbec4-d689-4d16-b520-21110b97d007-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx\" (UID: \"6eadbec4-d689-4d16-b520-21110b97d007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:27:00.261755 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.261734 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvm4\" (UniqueName: \"kubernetes.io/projected/6eadbec4-d689-4d16-b520-21110b97d007-kube-api-access-psvm4\") pod \"payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx\" (UID: \"6eadbec4-d689-4d16-b520-21110b97d007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:27:00.262084 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.262068 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n89k\" (UniqueName: \"kubernetes.io/projected/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-kube-api-access-7n89k\") pod \"payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn\" (UID: \"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:27:00.275997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.275978 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:27:00.284049 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.284030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:27:00.551719 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.551696 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx"] Sep 11 15:27:00.553510 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:27:00.553484 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eadbec4_d689_4d16_b520_21110b97d007.slice/crio-bc1be91c02be40eb2eddfacf8dd5278db68439f1fec2e98e323a10a52c29b064 WatchSource:0}: Error finding container bc1be91c02be40eb2eddfacf8dd5278db68439f1fec2e98e323a10a52c29b064: Status 404 returned error can't find the container with id bc1be91c02be40eb2eddfacf8dd5278db68439f1fec2e98e323a10a52c29b064 Sep 11 15:27:00.558266 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.558242 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn"] Sep 11 15:27:00.559962 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:27:00.559940 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae4adae_0d05_4a7e_b9f2_b646bc4b1bdc.slice/crio-ad1159a6b62677f52d085c07539e918ae27f07f5b61daa7d309c72847e259029 WatchSource:0}: Error finding container ad1159a6b62677f52d085c07539e918ae27f07f5b61daa7d309c72847e259029: Status 404 returned error can't find the container with id ad1159a6b62677f52d085c07539e918ae27f07f5b61daa7d309c72847e259029 Sep 11 15:27:00.799901 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.799854 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" event={"ID":"6eadbec4-d689-4d16-b520-21110b97d007","Type":"ContainerStarted","Data":"f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48"} Sep 11 15:27:00.800061 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.799920 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" event={"ID":"6eadbec4-d689-4d16-b520-21110b97d007","Type":"ContainerStarted","Data":"bc1be91c02be40eb2eddfacf8dd5278db68439f1fec2e98e323a10a52c29b064"} Sep 11 15:27:00.800061 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.799952 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:27:00.801912 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.801845 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" event={"ID":"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc","Type":"ContainerStarted","Data":"de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df"} Sep 11 15:27:00.801912 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.801890 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" event={"ID":"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc","Type":"ContainerStarted","Data":"ad1159a6b62677f52d085c07539e918ae27f07f5b61daa7d309c72847e259029"} Sep 11 15:27:00.802095 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.801970 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:27:00.815523 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.815379 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" podStartSLOduration=1.8153636469999999 podStartE2EDuration="1.815363647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:27:00.814659244 +0000 UTC m=+1102.859984225" watchObservedRunningTime="2026-09-11 15:27:00.815363647 +0000 UTC m=+1102.860688628" Sep 11 15:27:00.832372 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:00.832333 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" podStartSLOduration=1.832320115 podStartE2EDuration="1.832320115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:27:00.830575757 +0000 UTC m=+1102.875900737" watchObservedRunningTime="2026-09-11 15:27:00.832320115 +0000 UTC m=+1102.877645095" Sep 11 15:27:02.400171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.400132 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg"] Sep 11 15:27:02.403845 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.403823 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:27:02.406073 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.406048 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4a348b\"" Sep 11 15:27:02.413986 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.413966 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg"] Sep 11 15:27:02.444017 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.443974 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x"] Sep 11 15:27:02.447376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.447358 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:27:02.449806 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.449787 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4a348b-dockercfg-glq59\"" Sep 11 15:27:02.457653 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.457632 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x"] Sep 11 15:27:02.574484 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.574461 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhm7c\" (UniqueName: \"kubernetes.io/projected/87ab3b9b-a467-492b-ab28-6a4ce6c63752-kube-api-access-hhm7c\") pod \"payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x\" (UID: \"87ab3b9b-a467-492b-ab28-6a4ce6c63752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:27:02.574614 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.574506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7d0270-23ab-42dd-a63d-881a5b78cf56-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg\" (UID: \"6f7d0270-23ab-42dd-a63d-881a5b78cf56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:27:02.574614 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.574545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ab3b9b-a467-492b-ab28-6a4ce6c63752-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x\" (UID: \"87ab3b9b-a467-492b-ab28-6a4ce6c63752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:27:02.574709 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.574629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2lg\" (UniqueName: \"kubernetes.io/projected/6f7d0270-23ab-42dd-a63d-881a5b78cf56-kube-api-access-zj2lg\") pod \"payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg\" (UID: \"6f7d0270-23ab-42dd-a63d-881a5b78cf56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:27:02.675707 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.675641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2lg\" (UniqueName: \"kubernetes.io/projected/6f7d0270-23ab-42dd-a63d-881a5b78cf56-kube-api-access-zj2lg\") pod \"payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg\" (UID: \"6f7d0270-23ab-42dd-a63d-881a5b78cf56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:27:02.675707 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.675691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhm7c\" (UniqueName: \"kubernetes.io/projected/87ab3b9b-a467-492b-ab28-6a4ce6c63752-kube-api-access-hhm7c\") pod \"payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x\" (UID: \"87ab3b9b-a467-492b-ab28-6a4ce6c63752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:27:02.675920 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.675749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7d0270-23ab-42dd-a63d-881a5b78cf56-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg\" (UID: \"6f7d0270-23ab-42dd-a63d-881a5b78cf56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:27:02.675920 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.675858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ab3b9b-a467-492b-ab28-6a4ce6c63752-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x\" (UID: \"87ab3b9b-a467-492b-ab28-6a4ce6c63752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:27:02.676730 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.676708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ab3b9b-a467-492b-ab28-6a4ce6c63752-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x\" (UID: \"87ab3b9b-a467-492b-ab28-6a4ce6c63752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:27:02.676808 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.676708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7d0270-23ab-42dd-a63d-881a5b78cf56-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg\" (UID: \"6f7d0270-23ab-42dd-a63d-881a5b78cf56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:27:02.683518 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.683500 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhm7c\" (UniqueName: \"kubernetes.io/projected/87ab3b9b-a467-492b-ab28-6a4ce6c63752-kube-api-access-hhm7c\") pod \"payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x\" (UID: \"87ab3b9b-a467-492b-ab28-6a4ce6c63752\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:27:02.683594 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.683536 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2lg\" (UniqueName: \"kubernetes.io/projected/6f7d0270-23ab-42dd-a63d-881a5b78cf56-kube-api-access-zj2lg\") pod \"payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg\" (UID: \"6f7d0270-23ab-42dd-a63d-881a5b78cf56\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:27:02.715371 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.715350 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:27:02.757878 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.757853 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:27:02.987874 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:02.987841 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg"] Sep 11 15:27:03.039358 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:03.039335 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x"] Sep 11 15:27:03.041221 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:27:03.041199 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ab3b9b_a467_492b_ab28_6a4ce6c63752.slice/crio-66b7ca192aaa009790c1c37402af96e52368f2cc458d86ad18efc972c999b348 WatchSource:0}: Error finding container 66b7ca192aaa009790c1c37402af96e52368f2cc458d86ad18efc972c999b348: Status 404 returned error can't find the container with id 66b7ca192aaa009790c1c37402af96e52368f2cc458d86ad18efc972c999b348 Sep 11 15:27:03.822847 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:03.822813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" event={"ID":"87ab3b9b-a467-492b-ab28-6a4ce6c63752","Type":"ContainerStarted","Data":"8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620"} Sep 11 15:27:03.823268 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:03.822895 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" event={"ID":"87ab3b9b-a467-492b-ab28-6a4ce6c63752","Type":"ContainerStarted","Data":"66b7ca192aaa009790c1c37402af96e52368f2cc458d86ad18efc972c999b348"} Sep 11 15:27:03.823268 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:03.822924 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:27:03.824552 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:03.824526 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" event={"ID":"6f7d0270-23ab-42dd-a63d-881a5b78cf56","Type":"ContainerStarted","Data":"3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f"} Sep 11 15:27:03.824688 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:03.824574 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" event={"ID":"6f7d0270-23ab-42dd-a63d-881a5b78cf56","Type":"ContainerStarted","Data":"eb90927c903ee1302c7629259712cff001646f1fbb5be967d4dc4beb7ac6ea61"} Sep 11 15:27:03.824688 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:03.824623 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:27:03.871056 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:03.871010 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" podStartSLOduration=1.870996874 podStartE2EDuration="1.870996874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:27:03.868617877 +0000 UTC m=+1105.913942857" watchObservedRunningTime="2026-09-11 15:27:03.870996874 +0000 UTC m=+1105.916321854" Sep 11 15:27:03.889563 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:03.889520 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" podStartSLOduration=1.889497327 podStartE2EDuration="1.889497327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:27:03.887212257 +0000 UTC m=+1105.932537241" watchObservedRunningTime="2026-09-11 15:27:03.889497327 +0000 UTC m=+1105.934822307" Sep 11 15:27:11.810161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:11.810133 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:27:11.810621 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:11.810202 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:27:14.832186 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:14.832158 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:27:14.832555 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:14.832222 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:27:29.522905 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.522873 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl"] Sep 11 15:27:29.526310 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.526293 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:27:29.528390 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.528370 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-abba31\"" Sep 11 15:27:29.528732 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.528716 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-abba31-dockercfg-fgghq\"" Sep 11 15:27:29.538437 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.538413 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl"] Sep 11 15:27:29.547102 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.547080 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w"] Sep 11 15:27:29.550516 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.550496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:27:29.563767 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.563750 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w"] Sep 11 15:27:29.594992 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.594965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7006525e-e918-413f-88ff-05a518c5b8e6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w\" (UID: \"7006525e-e918-413f-88ff-05a518c5b8e6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:27:29.595096 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.594997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smmd\" (UniqueName: \"kubernetes.io/projected/7006525e-e918-413f-88ff-05a518c5b8e6-kube-api-access-9smmd\") pod \"payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w\" (UID: \"7006525e-e918-413f-88ff-05a518c5b8e6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:27:29.595096 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.595028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lwd\" (UniqueName: \"kubernetes.io/projected/3d083bca-313a-467c-b8ad-30ceab688d08-kube-api-access-n5lwd\") pod \"payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl\" (UID: \"3d083bca-313a-467c-b8ad-30ceab688d08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:27:29.595189 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.595110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d083bca-313a-467c-b8ad-30ceab688d08-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl\" (UID: \"3d083bca-313a-467c-b8ad-30ceab688d08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:27:29.695892 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.695858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7006525e-e918-413f-88ff-05a518c5b8e6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w\" (UID: \"7006525e-e918-413f-88ff-05a518c5b8e6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:27:29.696004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.695896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smmd\" (UniqueName: \"kubernetes.io/projected/7006525e-e918-413f-88ff-05a518c5b8e6-kube-api-access-9smmd\") pod \"payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w\" (UID: \"7006525e-e918-413f-88ff-05a518c5b8e6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:27:29.696004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.695932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lwd\" (UniqueName: \"kubernetes.io/projected/3d083bca-313a-467c-b8ad-30ceab688d08-kube-api-access-n5lwd\") pod \"payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl\" (UID: \"3d083bca-313a-467c-b8ad-30ceab688d08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:27:29.696004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.695953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d083bca-313a-467c-b8ad-30ceab688d08-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl\" (UID: \"3d083bca-313a-467c-b8ad-30ceab688d08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:27:29.696540 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.696521 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7006525e-e918-413f-88ff-05a518c5b8e6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w\" (UID: \"7006525e-e918-413f-88ff-05a518c5b8e6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:27:29.696670 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.696652 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d083bca-313a-467c-b8ad-30ceab688d08-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl\" (UID: \"3d083bca-313a-467c-b8ad-30ceab688d08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:27:29.707603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.707576 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smmd\" (UniqueName: \"kubernetes.io/projected/7006525e-e918-413f-88ff-05a518c5b8e6-kube-api-access-9smmd\") pod \"payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w\" (UID: \"7006525e-e918-413f-88ff-05a518c5b8e6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:27:29.708084 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.708065 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lwd\" (UniqueName: \"kubernetes.io/projected/3d083bca-313a-467c-b8ad-30ceab688d08-kube-api-access-n5lwd\") pod \"payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl\" (UID: \"3d083bca-313a-467c-b8ad-30ceab688d08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:27:29.836708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.836683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:27:29.863577 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:29.863552 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:27:30.112898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.112874 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl"] Sep 11 15:27:30.115107 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:27:30.115071 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d083bca_313a_467c_b8ad_30ceab688d08.slice/crio-b02e81a4af4a83e22869d758c545100af337db97ebb955c751b7851cb2606533 WatchSource:0}: Error finding container b02e81a4af4a83e22869d758c545100af337db97ebb955c751b7851cb2606533: Status 404 returned error can't find the container with id b02e81a4af4a83e22869d758c545100af337db97ebb955c751b7851cb2606533 Sep 11 15:27:30.145644 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.145623 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w"] Sep 11 15:27:30.149856 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:27:30.149829 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7006525e_e918_413f_88ff_05a518c5b8e6.slice/crio-34f93ee26ede3f4937591147a052b67ced9302b02f2b890d095a60fa9a1e6291 WatchSource:0}: Error finding container 34f93ee26ede3f4937591147a052b67ced9302b02f2b890d095a60fa9a1e6291: Status 404 returned error can't find the container with id 34f93ee26ede3f4937591147a052b67ced9302b02f2b890d095a60fa9a1e6291 Sep 11 15:27:30.819720 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.819684 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp"] Sep 11 15:27:30.823218 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.823200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:27:30.826741 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.826718 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7ee57f\"" Sep 11 15:27:30.826883 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.826750 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7ee57f-dockercfg-9r4r5\"" Sep 11 15:27:30.834808 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.834746 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp"] Sep 11 15:27:30.907796 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.907752 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvnxj\" (UniqueName: \"kubernetes.io/projected/f6a17136-c621-44d1-a042-5736ef91122e-kube-api-access-kvnxj\") pod \"payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp\" (UID: \"f6a17136-c621-44d1-a042-5736ef91122e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:27:30.907939 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.907826 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a17136-c621-44d1-a042-5736ef91122e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp\" (UID: \"f6a17136-c621-44d1-a042-5736ef91122e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:27:30.968846 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.968810 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" event={"ID":"3d083bca-313a-467c-b8ad-30ceab688d08","Type":"ContainerStarted","Data":"35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647"} Sep 11 15:27:30.968966 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.968874 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" event={"ID":"3d083bca-313a-467c-b8ad-30ceab688d08","Type":"ContainerStarted","Data":"b02e81a4af4a83e22869d758c545100af337db97ebb955c751b7851cb2606533"} Sep 11 15:27:30.968966 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.968903 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:27:30.970478 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.970453 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" event={"ID":"7006525e-e918-413f-88ff-05a518c5b8e6","Type":"ContainerStarted","Data":"eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c"} Sep 11 15:27:30.970580 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.970501 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" event={"ID":"7006525e-e918-413f-88ff-05a518c5b8e6","Type":"ContainerStarted","Data":"34f93ee26ede3f4937591147a052b67ced9302b02f2b890d095a60fa9a1e6291"} Sep 11 15:27:30.970580 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.970530 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:27:30.990141 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:30.990092 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" podStartSLOduration=1.9900793239999999 podStartE2EDuration="1.990079324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:27:30.987281738 +0000 UTC m=+1133.032606729" watchObservedRunningTime="2026-09-11 15:27:30.990079324 +0000 UTC m=+1133.035404304" Sep 11 15:27:31.005693 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:31.005630 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" podStartSLOduration=2.005622149 podStartE2EDuration="2.005622149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:27:31.004442006 +0000 UTC m=+1133.049766987" watchObservedRunningTime="2026-09-11 15:27:31.005622149 +0000 UTC m=+1133.050947128" Sep 11 15:27:31.009116 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:31.009094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvnxj\" (UniqueName: \"kubernetes.io/projected/f6a17136-c621-44d1-a042-5736ef91122e-kube-api-access-kvnxj\") pod \"payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp\" (UID: \"f6a17136-c621-44d1-a042-5736ef91122e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:27:31.009210 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:31.009129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a17136-c621-44d1-a042-5736ef91122e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp\" (UID: \"f6a17136-c621-44d1-a042-5736ef91122e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:27:31.009822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:31.009804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a17136-c621-44d1-a042-5736ef91122e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp\" (UID: \"f6a17136-c621-44d1-a042-5736ef91122e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:27:31.017787 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:31.017741 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvnxj\" (UniqueName: \"kubernetes.io/projected/f6a17136-c621-44d1-a042-5736ef91122e-kube-api-access-kvnxj\") pod \"payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp\" (UID: \"f6a17136-c621-44d1-a042-5736ef91122e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:27:31.135237 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:31.135217 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:27:31.407036 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:31.406997 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp"] Sep 11 15:27:31.407707 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:27:31.407678 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a17136_c621_44d1_a042_5736ef91122e.slice/crio-608736ee21f6783b50dffdb319b1608d887535751e3afb531d6d659c209f1610 WatchSource:0}: Error finding container 608736ee21f6783b50dffdb319b1608d887535751e3afb531d6d659c209f1610: Status 404 returned error can't find the container with id 608736ee21f6783b50dffdb319b1608d887535751e3afb531d6d659c209f1610 Sep 11 15:27:31.979837 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:31.979802 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" event={"ID":"f6a17136-c621-44d1-a042-5736ef91122e","Type":"ContainerStarted","Data":"4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed"} Sep 11 15:27:31.980247 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:31.979865 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" event={"ID":"f6a17136-c621-44d1-a042-5736ef91122e","Type":"ContainerStarted","Data":"608736ee21f6783b50dffdb319b1608d887535751e3afb531d6d659c209f1610"} Sep 11 15:27:31.997193 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:31.997149 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" podStartSLOduration=1.997132898 podStartE2EDuration="1.997132898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:27:31.99555821 +0000 UTC m=+1134.040883194" watchObservedRunningTime="2026-09-11 15:27:31.997132898 +0000 UTC m=+1134.042457878" Sep 11 15:27:32.985100 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:32.985072 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:27:41.982469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:41.982437 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:27:41.982992 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:41.982523 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:27:43.993729 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:43.993699 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:27:54.682656 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.682626 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r"] Sep 11 15:27:54.688078 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.688058 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.691240 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.691213 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-397e40\"/\"kube-root-ca.crt\"" Sep 11 15:27:54.691240 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.691237 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-397e40\"/\"auth-test-model-397e40-kserve-self-signed-certs\"" Sep 11 15:27:54.691422 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.691213 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-397e40\"/\"openshift-service-ca.crt\"" Sep 11 15:27:54.691422 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.691226 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-397e40\"/\"default-dockercfg-bjs5g\"" Sep 11 15:27:54.697190 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.697168 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r"] Sep 11 15:27:54.816428 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.816399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-dshm\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.816428 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.816430 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-model-cache\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.816598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.816449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8ld\" (UniqueName: \"kubernetes.io/projected/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-kube-api-access-zh8ld\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.816598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.816467 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tls-certs\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.816598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.816542 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-home\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.816598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.816576 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tmp-dir\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.917422 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.917390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-dshm\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.917422 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.917423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-model-cache\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.917593 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.917443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8ld\" (UniqueName: \"kubernetes.io/projected/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-kube-api-access-zh8ld\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.917593 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.917556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tls-certs\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.917679 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.917630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-home\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.917679 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.917672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tmp-dir\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.917840 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.917818 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-model-cache\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.917998 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.917977 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-home\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.918049 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.918011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tmp-dir\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.919825 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.919805 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-dshm\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.920276 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.920255 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tls-certs\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:54.925152 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:54.925131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8ld\" (UniqueName: \"kubernetes.io/projected/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-kube-api-access-zh8ld\") pod \"auth-test-model-397e40-kserve-f6b97d499-6t78r\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:55.013955 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:55.013903 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:27:55.283898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:55.282978 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r"] Sep 11 15:27:55.287432 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:27:55.287398 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb64fc4d_4ee9_4025_9996_54b3051c7e5c.slice/crio-f838b97179c9f772410c6161c31b75562944713318bdb85184b4b413a6b1658f WatchSource:0}: Error finding container f838b97179c9f772410c6161c31b75562944713318bdb85184b4b413a6b1658f: Status 404 returned error can't find the container with id f838b97179c9f772410c6161c31b75562944713318bdb85184b4b413a6b1658f Sep 11 15:27:56.107629 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:56.107585 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" event={"ID":"fb64fc4d-4ee9-4025-9996-54b3051c7e5c","Type":"ContainerStarted","Data":"699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af"} Sep 11 15:27:56.108015 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:56.107639 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" event={"ID":"fb64fc4d-4ee9-4025-9996-54b3051c7e5c","Type":"ContainerStarted","Data":"f838b97179c9f772410c6161c31b75562944713318bdb85184b4b413a6b1658f"} Sep 11 15:27:56.124480 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:56.124433 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" podStartSLOduration=2.124418823 podStartE2EDuration="2.124418823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:27:56.123252767 +0000 UTC m=+1158.168577751" watchObservedRunningTime="2026-09-11 15:27:56.124418823 +0000 UTC m=+1158.169743802" Sep 11 15:27:59.375603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:59.375569 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-52d4cb"] Sep 11 15:27:59.379060 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:59.379043 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-52d4cb" Sep 11 15:27:59.382922 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:59.382900 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6mbx9\"" Sep 11 15:27:59.396198 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:59.396168 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-52d4cb"] Sep 11 15:27:59.451227 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:59.451199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwhsk\" (UniqueName: \"kubernetes.io/projected/363eb51f-916e-48b1-9bb6-7d64a8a0d37e-kube-api-access-nwhsk\") pod \"test-curl-7337-52d4cb\" (UID: \"363eb51f-916e-48b1-9bb6-7d64a8a0d37e\") " pod="opendatahub/test-curl-7337-52d4cb" Sep 11 15:27:59.552194 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:59.552144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwhsk\" (UniqueName: \"kubernetes.io/projected/363eb51f-916e-48b1-9bb6-7d64a8a0d37e-kube-api-access-nwhsk\") pod \"test-curl-7337-52d4cb\" (UID: \"363eb51f-916e-48b1-9bb6-7d64a8a0d37e\") " pod="opendatahub/test-curl-7337-52d4cb" Sep 11 15:27:59.560388 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:59.560368 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwhsk\" (UniqueName: \"kubernetes.io/projected/363eb51f-916e-48b1-9bb6-7d64a8a0d37e-kube-api-access-nwhsk\") pod \"test-curl-7337-52d4cb\" (UID: \"363eb51f-916e-48b1-9bb6-7d64a8a0d37e\") " pod="opendatahub/test-curl-7337-52d4cb" Sep 11 15:27:59.688213 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:59.688130 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-52d4cb" Sep 11 15:27:59.956631 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:27:59.956606 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-52d4cb"] Sep 11 15:27:59.958600 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:27:59.958572 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod363eb51f_916e_48b1_9bb6_7d64a8a0d37e.slice/crio-ac126c6fbebb91eb66354da629f88d1fc35796718dc838b981d7e6370ef0e6b5 WatchSource:0}: Error finding container ac126c6fbebb91eb66354da629f88d1fc35796718dc838b981d7e6370ef0e6b5: Status 404 returned error can't find the container with id ac126c6fbebb91eb66354da629f88d1fc35796718dc838b981d7e6370ef0e6b5 Sep 11 15:28:00.131045 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.131014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-52d4cb" event={"ID":"363eb51f-916e-48b1-9bb6-7d64a8a0d37e","Type":"ContainerStarted","Data":"7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f"} Sep 11 15:28:00.131151 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.131062 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-52d4cb" event={"ID":"363eb51f-916e-48b1-9bb6-7d64a8a0d37e","Type":"ContainerStarted","Data":"ac126c6fbebb91eb66354da629f88d1fc35796718dc838b981d7e6370ef0e6b5"} Sep 11 15:28:00.147616 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.147568 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7337-52d4cb" podStartSLOduration=1.147552069 podStartE2EDuration="1.147552069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:28:00.1456048 +0000 UTC m=+1162.190929780" watchObservedRunningTime="2026-09-11 15:28:00.147552069 +0000 UTC m=+1162.192877050" Sep 11 15:28:00.717510 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.717481 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-52d4cb"] Sep 11 15:28:00.721627 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.721601 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-52d4cb"] Sep 11 15:28:00.829378 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.829350 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-d9f0f5"] Sep 11 15:28:00.829734 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.829722 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="363eb51f-916e-48b1-9bb6-7d64a8a0d37e" containerName="test-curl-7337-52d4cb" Sep 11 15:28:00.829803 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.829736 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="363eb51f-916e-48b1-9bb6-7d64a8a0d37e" containerName="test-curl-7337-52d4cb" Sep 11 15:28:00.829840 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.829829 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="363eb51f-916e-48b1-9bb6-7d64a8a0d37e" containerName="test-curl-7337-52d4cb" Sep 11 15:28:00.833163 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.833148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-d9f0f5" Sep 11 15:28:00.840636 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.840615 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-d9f0f5"] Sep 11 15:28:00.862352 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.862328 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8fr9\" (UniqueName: \"kubernetes.io/projected/19cea728-448a-4f84-86b8-60ac2b7ed784-kube-api-access-q8fr9\") pod \"test-curl-7337-d9f0f5\" (UID: \"19cea728-448a-4f84-86b8-60ac2b7ed784\") " pod="opendatahub/test-curl-7337-d9f0f5" Sep 11 15:28:00.963322 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.963289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8fr9\" (UniqueName: \"kubernetes.io/projected/19cea728-448a-4f84-86b8-60ac2b7ed784-kube-api-access-q8fr9\") pod \"test-curl-7337-d9f0f5\" (UID: \"19cea728-448a-4f84-86b8-60ac2b7ed784\") " pod="opendatahub/test-curl-7337-d9f0f5" Sep 11 15:28:00.971723 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:00.971667 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8fr9\" (UniqueName: \"kubernetes.io/projected/19cea728-448a-4f84-86b8-60ac2b7ed784-kube-api-access-q8fr9\") pod \"test-curl-7337-d9f0f5\" (UID: \"19cea728-448a-4f84-86b8-60ac2b7ed784\") " pod="opendatahub/test-curl-7337-d9f0f5" Sep 11 15:28:01.142717 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:01.142687 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7337-52d4cb" container="test-curl-7337-52d4cb" Sep 11 15:28:01.142905 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:01.142890 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-d9f0f5" Sep 11 15:28:01.416422 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:01.416400 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-d9f0f5"] Sep 11 15:28:01.418456 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:28:01.418425 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19cea728_448a_4f84_86b8_60ac2b7ed784.slice/crio-463a46cc89e5157cd847613892595d47714113a7efd941735cc854890da2fe6e WatchSource:0}: Error finding container 463a46cc89e5157cd847613892595d47714113a7efd941735cc854890da2fe6e: Status 404 returned error can't find the container with id 463a46cc89e5157cd847613892595d47714113a7efd941735cc854890da2fe6e Sep 11 15:28:02.149142 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.149109 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-d9f0f5" event={"ID":"19cea728-448a-4f84-86b8-60ac2b7ed784","Type":"ContainerStarted","Data":"423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456"} Sep 11 15:28:02.149142 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.149152 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-d9f0f5" event={"ID":"19cea728-448a-4f84-86b8-60ac2b7ed784","Type":"ContainerStarted","Data":"463a46cc89e5157cd847613892595d47714113a7efd941735cc854890da2fe6e"} Sep 11 15:28:02.149671 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.149195 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7337-52d4cb" podUID="363eb51f-916e-48b1-9bb6-7d64a8a0d37e" containerName="test-curl-7337-52d4cb" containerID="cri-o://7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f" gracePeriod=2 Sep 11 15:28:02.165016 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.164971 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7337-d9f0f5" podStartSLOduration=2.164956998 podStartE2EDuration="2.164956998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:28:02.163755179 +0000 UTC m=+1164.209080160" watchObservedRunningTime="2026-09-11 15:28:02.164956998 +0000 UTC m=+1164.210281979" Sep 11 15:28:02.166698 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:02.166670 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-52d4cb\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="363eb51f-916e-48b1-9bb6-7d64a8a0d37e" pod="opendatahub/test-curl-7337-52d4cb" Sep 11 15:28:02.168359 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:02.168335 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-52d4cb\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="363eb51f-916e-48b1-9bb6-7d64a8a0d37e" pod="opendatahub/test-curl-7337-52d4cb" Sep 11 15:28:02.774206 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.774176 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-d9f0f5"] Sep 11 15:28:02.778194 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.778168 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-d9f0f5"] Sep 11 15:28:02.904545 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.904515 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-757a40"] Sep 11 15:28:02.904945 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.904932 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cea728-448a-4f84-86b8-60ac2b7ed784" containerName="test-curl-7337-d9f0f5" Sep 11 15:28:02.905008 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.904947 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cea728-448a-4f84-86b8-60ac2b7ed784" containerName="test-curl-7337-d9f0f5" Sep 11 15:28:02.905046 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.905021 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="19cea728-448a-4f84-86b8-60ac2b7ed784" containerName="test-curl-7337-d9f0f5" Sep 11 15:28:02.908292 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.908276 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-757a40" Sep 11 15:28:02.913886 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.913865 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-757a40"] Sep 11 15:28:02.980142 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:02.980113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz4t2\" (UniqueName: \"kubernetes.io/projected/6e611dd2-4a1d-4084-a8d9-8293d1b51c87-kube-api-access-lz4t2\") pod \"test-curl-7337-757a40\" (UID: \"6e611dd2-4a1d-4084-a8d9-8293d1b51c87\") " pod="opendatahub/test-curl-7337-757a40" Sep 11 15:28:03.081150 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:03.081121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz4t2\" (UniqueName: \"kubernetes.io/projected/6e611dd2-4a1d-4084-a8d9-8293d1b51c87-kube-api-access-lz4t2\") pod \"test-curl-7337-757a40\" (UID: \"6e611dd2-4a1d-4084-a8d9-8293d1b51c87\") " pod="opendatahub/test-curl-7337-757a40" Sep 11 15:28:03.089234 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:03.089208 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz4t2\" (UniqueName: \"kubernetes.io/projected/6e611dd2-4a1d-4084-a8d9-8293d1b51c87-kube-api-access-lz4t2\") pod \"test-curl-7337-757a40\" (UID: \"6e611dd2-4a1d-4084-a8d9-8293d1b51c87\") " pod="opendatahub/test-curl-7337-757a40" Sep 11 15:28:03.154881 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:03.154829 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7337-d9f0f5" container="test-curl-7337-d9f0f5" Sep 11 15:28:03.219066 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:03.219041 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-757a40" Sep 11 15:28:03.487811 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:03.487789 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-757a40"] Sep 11 15:28:03.489971 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:28:03.489940 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e611dd2_4a1d_4084_a8d9_8293d1b51c87.slice/crio-e1545ff35707777fd5494d513f0bafd47590c070952c9abb0108b77e4287ddb1 WatchSource:0}: Error finding container e1545ff35707777fd5494d513f0bafd47590c070952c9abb0108b77e4287ddb1: Status 404 returned error can't find the container with id e1545ff35707777fd5494d513f0bafd47590c070952c9abb0108b77e4287ddb1 Sep 11 15:28:04.161896 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.161865 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-757a40" event={"ID":"6e611dd2-4a1d-4084-a8d9-8293d1b51c87","Type":"ContainerStarted","Data":"ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf"} Sep 11 15:28:04.162267 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.161910 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-757a40" event={"ID":"6e611dd2-4a1d-4084-a8d9-8293d1b51c87","Type":"ContainerStarted","Data":"e1545ff35707777fd5494d513f0bafd47590c070952c9abb0108b77e4287ddb1"} Sep 11 15:28:04.162267 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.162031 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7337-d9f0f5" podUID="19cea728-448a-4f84-86b8-60ac2b7ed784" containerName="test-curl-7337-d9f0f5" containerID="cri-o://423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456" gracePeriod=2 Sep 11 15:28:04.177473 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.177416 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7337-757a40" podStartSLOduration=2.177400083 podStartE2EDuration="2.177400083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:28:04.175700411 +0000 UTC m=+1166.221025417" watchObservedRunningTime="2026-09-11 15:28:04.177400083 +0000 UTC m=+1166.222725063" Sep 11 15:28:04.177584 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:04.177520 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-d9f0f5\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="19cea728-448a-4f84-86b8-60ac2b7ed784" pod="opendatahub/test-curl-7337-d9f0f5" Sep 11 15:28:04.334281 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.334256 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-52d4cb" Sep 11 15:28:04.336077 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:04.336053 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-d9f0f5\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="19cea728-448a-4f84-86b8-60ac2b7ed784" pod="opendatahub/test-curl-7337-d9f0f5" Sep 11 15:28:04.391060 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.391039 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwhsk\" (UniqueName: \"kubernetes.io/projected/363eb51f-916e-48b1-9bb6-7d64a8a0d37e-kube-api-access-nwhsk\") pod \"363eb51f-916e-48b1-9bb6-7d64a8a0d37e\" (UID: \"363eb51f-916e-48b1-9bb6-7d64a8a0d37e\") " Sep 11 15:28:04.392999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.392976 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363eb51f-916e-48b1-9bb6-7d64a8a0d37e-kube-api-access-nwhsk" (OuterVolumeSpecName: "kube-api-access-nwhsk") pod "363eb51f-916e-48b1-9bb6-7d64a8a0d37e" (UID: "363eb51f-916e-48b1-9bb6-7d64a8a0d37e"). InnerVolumeSpecName "kube-api-access-nwhsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:28:04.492212 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.492156 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwhsk\" (UniqueName: \"kubernetes.io/projected/363eb51f-916e-48b1-9bb6-7d64a8a0d37e-kube-api-access-nwhsk\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:28:04.493080 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.493059 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="363eb51f-916e-48b1-9bb6-7d64a8a0d37e" path="/var/lib/kubelet/pods/363eb51f-916e-48b1-9bb6-7d64a8a0d37e/volumes" Sep 11 15:28:04.746033 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.745954 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-757a40"] Sep 11 15:28:04.749828 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.749805 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-757a40"] Sep 11 15:28:04.860589 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.860566 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-8ce7fd"] Sep 11 15:28:04.861044 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.861029 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e611dd2-4a1d-4084-a8d9-8293d1b51c87" containerName="test-curl-7337-757a40" Sep 11 15:28:04.861098 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.861045 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e611dd2-4a1d-4084-a8d9-8293d1b51c87" containerName="test-curl-7337-757a40" Sep 11 15:28:04.861137 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.861110 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e611dd2-4a1d-4084-a8d9-8293d1b51c87" containerName="test-curl-7337-757a40" Sep 11 15:28:04.864184 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.864167 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-8ce7fd" Sep 11 15:28:04.867765 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.867731 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-8ce7fd"] Sep 11 15:28:04.903476 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:04.903448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55fs\" (UniqueName: \"kubernetes.io/projected/4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e-kube-api-access-k55fs\") pod \"test-curl-7337-8ce7fd\" (UID: \"4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e\") " pod="opendatahub/test-curl-7337-8ce7fd" Sep 11 15:28:05.004802 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.004727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55fs\" (UniqueName: \"kubernetes.io/projected/4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e-kube-api-access-k55fs\") pod \"test-curl-7337-8ce7fd\" (UID: \"4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e\") " pod="opendatahub/test-curl-7337-8ce7fd" Sep 11 15:28:05.012548 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.012522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55fs\" (UniqueName: \"kubernetes.io/projected/4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e-kube-api-access-k55fs\") pod \"test-curl-7337-8ce7fd\" (UID: \"4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e\") " pod="opendatahub/test-curl-7337-8ce7fd" Sep 11 15:28:05.014409 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.014384 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:28:05.014521 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.014421 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:28:05.027413 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.027397 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:28:05.168485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.168452 2573 generic.go:356] "Generic (PLEG): container finished" podID="363eb51f-916e-48b1-9bb6-7d64a8a0d37e" containerID="7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f" exitCode=137 Sep 11 15:28:05.168837 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.168505 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-52d4cb" Sep 11 15:28:05.168837 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.168546 2573 scope.go:117] "RemoveContainer" containerID="7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f" Sep 11 15:28:05.168837 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:05.168805 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7337-757a40" container="test-curl-7337-757a40" Sep 11 15:28:05.176405 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.176382 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-8ce7fd" Sep 11 15:28:05.178372 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.178357 2573 scope.go:117] "RemoveContainer" containerID="7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f" Sep 11 15:28:05.178654 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:05.178630 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f\": container with ID starting with 7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f not found: ID does not exist" containerID="7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f" Sep 11 15:28:05.178730 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.178666 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f"} err="failed to get container status \"7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f\": rpc error: code = NotFound desc = could not find container \"7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f\": container with ID starting with 7fc5e2431fa93f49062686626f92066ff90eda8f1f858490438cbb3b5008a82f not found: ID does not exist" Sep 11 15:28:05.181628 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.181602 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:28:05.451393 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:05.451357 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-8ce7fd"] Sep 11 15:28:05.453511 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:28:05.453482 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4113b1eb_6bc2_4769_88a1_7e8b96c6ab6e.slice/crio-b0650fca896d861fd99f9b8ef9937c6c489e71da913b11530688b9ea626b66df WatchSource:0}: Error finding container b0650fca896d861fd99f9b8ef9937c6c489e71da913b11530688b9ea626b66df: Status 404 returned error can't find the container with id b0650fca896d861fd99f9b8ef9937c6c489e71da913b11530688b9ea626b66df Sep 11 15:28:06.176126 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.176093 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-8ce7fd" event={"ID":"4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e","Type":"ContainerStarted","Data":"c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a"} Sep 11 15:28:06.176454 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.176142 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-8ce7fd" event={"ID":"4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e","Type":"ContainerStarted","Data":"b0650fca896d861fd99f9b8ef9937c6c489e71da913b11530688b9ea626b66df"} Sep 11 15:28:06.177589 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.177564 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7337-757a40" podUID="6e611dd2-4a1d-4084-a8d9-8293d1b51c87" containerName="test-curl-7337-757a40" containerID="cri-o://ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf" gracePeriod=2 Sep 11 15:28:06.190903 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:06.190875 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-757a40\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="6e611dd2-4a1d-4084-a8d9-8293d1b51c87" pod="opendatahub/test-curl-7337-757a40" Sep 11 15:28:06.191210 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.191174 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7337-8ce7fd" podStartSLOduration=2.191161294 podStartE2EDuration="2.191161294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:28:06.189122661 +0000 UTC m=+1168.234447637" watchObservedRunningTime="2026-09-11 15:28:06.191161294 +0000 UTC m=+1168.236486274" Sep 11 15:28:06.343693 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.343671 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-d9f0f5" Sep 11 15:28:06.345630 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:06.345604 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-757a40\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="6e611dd2-4a1d-4084-a8d9-8293d1b51c87" pod="opendatahub/test-curl-7337-757a40" Sep 11 15:28:06.417798 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.417718 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8fr9\" (UniqueName: \"kubernetes.io/projected/19cea728-448a-4f84-86b8-60ac2b7ed784-kube-api-access-q8fr9\") pod \"19cea728-448a-4f84-86b8-60ac2b7ed784\" (UID: \"19cea728-448a-4f84-86b8-60ac2b7ed784\") " Sep 11 15:28:06.419670 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.419649 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cea728-448a-4f84-86b8-60ac2b7ed784-kube-api-access-q8fr9" (OuterVolumeSpecName: "kube-api-access-q8fr9") pod "19cea728-448a-4f84-86b8-60ac2b7ed784" (UID: "19cea728-448a-4f84-86b8-60ac2b7ed784"). InnerVolumeSpecName "kube-api-access-q8fr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:28:06.492831 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.492805 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19cea728-448a-4f84-86b8-60ac2b7ed784" path="/var/lib/kubelet/pods/19cea728-448a-4f84-86b8-60ac2b7ed784/volumes" Sep 11 15:28:06.519020 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.518999 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8fr9\" (UniqueName: \"kubernetes.io/projected/19cea728-448a-4f84-86b8-60ac2b7ed784-kube-api-access-q8fr9\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:28:06.779396 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.779333 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-8ce7fd"] Sep 11 15:28:06.783394 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:06.783370 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-8ce7fd"] Sep 11 15:28:07.183906 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:07.183873 2573 generic.go:356] "Generic (PLEG): container finished" podID="19cea728-448a-4f84-86b8-60ac2b7ed784" containerID="423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456" exitCode=137 Sep 11 15:28:07.184343 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:07.183913 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-d9f0f5" Sep 11 15:28:07.184343 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:07.183969 2573 scope.go:117] "RemoveContainer" containerID="423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456" Sep 11 15:28:07.184343 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:07.184197 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7337-8ce7fd" container="test-curl-7337-8ce7fd" Sep 11 15:28:07.192453 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:07.192439 2573 scope.go:117] "RemoveContainer" containerID="423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456" Sep 11 15:28:07.192693 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:07.192677 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456\": container with ID starting with 423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456 not found: ID does not exist" containerID="423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456" Sep 11 15:28:07.192744 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:07.192700 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456"} err="failed to get container status \"423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456\": rpc error: code = NotFound desc = could not find container \"423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456\": container with ID starting with 423858db0ac9a5194dddf2003e0c486c2d2054b80b9b78a2d8c38cc107ff7456 not found: ID does not exist" Sep 11 15:28:08.190856 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:08.190671 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7337-8ce7fd" podUID="4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e" containerName="test-curl-7337-8ce7fd" containerID="cri-o://c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a" gracePeriod=2 Sep 11 15:28:08.192684 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:08.192652 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-8ce7fd\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e" pod="opendatahub/test-curl-7337-8ce7fd" Sep 11 15:28:08.366363 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:08.366340 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-757a40" Sep 11 15:28:08.368582 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:08.368555 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-8ce7fd\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e" pod="opendatahub/test-curl-7337-8ce7fd" Sep 11 15:28:08.434531 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:08.434464 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz4t2\" (UniqueName: \"kubernetes.io/projected/6e611dd2-4a1d-4084-a8d9-8293d1b51c87-kube-api-access-lz4t2\") pod \"6e611dd2-4a1d-4084-a8d9-8293d1b51c87\" (UID: \"6e611dd2-4a1d-4084-a8d9-8293d1b51c87\") " Sep 11 15:28:08.436603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:08.436577 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e611dd2-4a1d-4084-a8d9-8293d1b51c87-kube-api-access-lz4t2" (OuterVolumeSpecName: "kube-api-access-lz4t2") pod "6e611dd2-4a1d-4084-a8d9-8293d1b51c87" (UID: "6e611dd2-4a1d-4084-a8d9-8293d1b51c87"). InnerVolumeSpecName "kube-api-access-lz4t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:28:08.494193 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:08.494155 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7337-8ce7fd\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e" pod="opendatahub/test-curl-7337-8ce7fd" Sep 11 15:28:08.499057 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:08.499035 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e611dd2-4a1d-4084-a8d9-8293d1b51c87" path="/var/lib/kubelet/pods/6e611dd2-4a1d-4084-a8d9-8293d1b51c87/volumes" Sep 11 15:28:08.536024 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:08.536002 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz4t2\" (UniqueName: \"kubernetes.io/projected/6e611dd2-4a1d-4084-a8d9-8293d1b51c87-kube-api-access-lz4t2\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:28:09.197209 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:09.197178 2573 generic.go:356] "Generic (PLEG): container finished" podID="6e611dd2-4a1d-4084-a8d9-8293d1b51c87" containerID="ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf" exitCode=137 Sep 11 15:28:09.197586 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:09.197233 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-757a40" Sep 11 15:28:09.197586 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:09.197267 2573 scope.go:117] "RemoveContainer" containerID="ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf" Sep 11 15:28:09.205977 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:09.205962 2573 scope.go:117] "RemoveContainer" containerID="ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf" Sep 11 15:28:09.206264 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:09.206247 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf\": container with ID starting with ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf not found: ID does not exist" containerID="ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf" Sep 11 15:28:09.206318 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:09.206271 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf"} err="failed to get container status \"ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf\": rpc error: code = NotFound desc = could not find container \"ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf\": container with ID starting with ba2609edaaa1689adc11cccb8f8ece404b470998ece185acf4816c084618fbcf not found: ID does not exist" Sep 11 15:28:10.340702 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:10.340681 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-8ce7fd" Sep 11 15:28:10.451216 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:10.451151 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k55fs\" (UniqueName: \"kubernetes.io/projected/4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e-kube-api-access-k55fs\") pod \"4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e\" (UID: \"4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e\") " Sep 11 15:28:10.453337 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:10.453311 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e-kube-api-access-k55fs" (OuterVolumeSpecName: "kube-api-access-k55fs") pod "4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e" (UID: "4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e"). InnerVolumeSpecName "kube-api-access-k55fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:28:10.493166 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:10.493143 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e" path="/var/lib/kubelet/pods/4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e/volumes" Sep 11 15:28:10.552201 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:10.552173 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k55fs\" (UniqueName: \"kubernetes.io/projected/4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e-kube-api-access-k55fs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:28:11.211172 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:11.211139 2573 generic.go:356] "Generic (PLEG): container finished" podID="4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e" containerID="c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a" exitCode=137 Sep 11 15:28:11.211341 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:11.211181 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-8ce7fd" Sep 11 15:28:11.211341 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:11.211240 2573 scope.go:117] "RemoveContainer" containerID="c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a" Sep 11 15:28:11.219971 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:11.219950 2573 scope.go:117] "RemoveContainer" containerID="c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a" Sep 11 15:28:11.220279 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:28:11.220259 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a\": container with ID starting with c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a not found: ID does not exist" containerID="c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a" Sep 11 15:28:11.220352 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:11.220289 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a"} err="failed to get container status \"c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a\": rpc error: code = NotFound desc = could not find container \"c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a\": container with ID starting with c9a3757f0f72c5ef53584228146151f4532142493a892351137e1b0d53427e9a not found: ID does not exist" Sep 11 15:28:25.984644 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:25.984608 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6"] Sep 11 15:28:25.985273 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:25.985249 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e" containerName="test-curl-7337-8ce7fd" Sep 11 15:28:25.985372 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:25.985275 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e" containerName="test-curl-7337-8ce7fd" Sep 11 15:28:25.985425 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:25.985405 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4113b1eb-6bc2-4769-88a1-7e8b96c6ab6e" containerName="test-curl-7337-8ce7fd" Sep 11 15:28:25.990257 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:25.990231 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:25.992958 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:25.992932 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-abba31\"/\"kube-root-ca.crt\"" Sep 11 15:28:25.993803 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:25.993767 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-abba31\"/\"openshift-service-ca.crt\"" Sep 11 15:28:25.993896 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:25.993812 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-abba31\"/\"auth-test-model-abba31-kserve-self-signed-certs\"" Sep 11 15:28:25.993975 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:25.993957 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-abba31\"/\"default-dockercfg-qdbqn\"" Sep 11 15:28:25.998757 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:25.997870 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6"] Sep 11 15:28:26.077042 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.077015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-home\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.077148 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.077080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-dshm\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.077148 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.077128 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-tmp-dir\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.077256 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.077167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde3d533-8dba-46f1-9281-a00f9586dcca-tls-certs\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.077256 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.077229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-model-cache\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.077375 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.077300 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r47dc\" (UniqueName: \"kubernetes.io/projected/dde3d533-8dba-46f1-9281-a00f9586dcca-kube-api-access-r47dc\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.177883 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.177853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde3d533-8dba-46f1-9281-a00f9586dcca-tls-certs\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.177883 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.177887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-model-cache\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.178038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.177907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r47dc\" (UniqueName: \"kubernetes.io/projected/dde3d533-8dba-46f1-9281-a00f9586dcca-kube-api-access-r47dc\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.178038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.177939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-home\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.178038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.177965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-dshm\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.178038 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.178004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-tmp-dir\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.178300 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.178279 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-model-cache\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.178365 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.178351 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-tmp-dir\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.178405 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.178389 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-home\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.180346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.180318 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-dshm\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.180616 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.180598 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde3d533-8dba-46f1-9281-a00f9586dcca-tls-certs\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.186344 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.186324 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r47dc\" (UniqueName: \"kubernetes.io/projected/dde3d533-8dba-46f1-9281-a00f9586dcca-kube-api-access-r47dc\") pod \"auth-test-model-abba31-kserve-848fbdbd59-9scs6\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.302476 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.302423 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:26.572112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:26.572083 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6"] Sep 11 15:28:26.573721 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:28:26.573689 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddde3d533_8dba_46f1_9281_a00f9586dcca.slice/crio-2fc95fa3f45b7e52ba14dc7931e8f4194f7abc0626f04283ae585b1bf6779f63 WatchSource:0}: Error finding container 2fc95fa3f45b7e52ba14dc7931e8f4194f7abc0626f04283ae585b1bf6779f63: Status 404 returned error can't find the container with id 2fc95fa3f45b7e52ba14dc7931e8f4194f7abc0626f04283ae585b1bf6779f63 Sep 11 15:28:27.299168 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:27.299136 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" event={"ID":"dde3d533-8dba-46f1-9281-a00f9586dcca","Type":"ContainerStarted","Data":"045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8"} Sep 11 15:28:27.299573 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:27.299181 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" event={"ID":"dde3d533-8dba-46f1-9281-a00f9586dcca","Type":"ContainerStarted","Data":"2fc95fa3f45b7e52ba14dc7931e8f4194f7abc0626f04283ae585b1bf6779f63"} Sep 11 15:28:27.318148 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:27.318097 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" podStartSLOduration=2.31808137 podStartE2EDuration="2.31808137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:28:27.317058675 +0000 UTC m=+1189.362383654" watchObservedRunningTime="2026-09-11 15:28:27.31808137 +0000 UTC m=+1189.363406376" Sep 11 15:28:36.303256 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:36.303220 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:36.303719 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:36.303448 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:36.315950 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:36.315929 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:36.357845 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:36.357822 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:28:38.471573 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:38.471536 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:28:38.474322 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:38.474297 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:28:38.599567 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:38.599540 2573 scope.go:117] "RemoveContainer" containerID="67af3366fc2d6996360d33f26c7b340597d352c6ef3b450907fd74350e3b8ccf" Sep 11 15:28:52.581987 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.581953 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q"] Sep 11 15:28:52.587291 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.587269 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.590508 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.590487 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dcdcc9ba\"/\"e2e-lifecycle-model-dcdcc9ba-kserve-self-signed-certs\"" Sep 11 15:28:52.590641 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.590502 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dcdcc9ba\"/\"default-dockercfg-hzr6p\"" Sep 11 15:28:52.590641 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.590503 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dcdcc9ba\"/\"openshift-service-ca.crt\"" Sep 11 15:28:52.590641 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.590533 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dcdcc9ba\"/\"kube-root-ca.crt\"" Sep 11 15:28:52.596741 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.596717 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q"] Sep 11 15:28:52.697539 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.697515 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9pk\" (UniqueName: \"kubernetes.io/projected/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-kube-api-access-ls9pk\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.697690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.697554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tmp-dir\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.697690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.697591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-model-cache\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.697690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.697639 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-dshm\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.697898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.697742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tls-certs\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.697898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.697805 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-home\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.798994 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.798967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tls-certs\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.799116 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.799002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-home\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.799116 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.799026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9pk\" (UniqueName: \"kubernetes.io/projected/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-kube-api-access-ls9pk\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.799116 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.799046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tmp-dir\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.799116 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.799078 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-model-cache\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.799116 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.799105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-dshm\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.799445 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.799425 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-home\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.799546 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.799531 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tmp-dir\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.799605 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.799585 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-model-cache\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.801528 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.801505 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-dshm\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.801719 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.801702 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tls-certs\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.806949 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.806927 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9pk\" (UniqueName: \"kubernetes.io/projected/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-kube-api-access-ls9pk\") pod \"e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:52.898111 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:52.898043 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:28:53.171876 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:53.171848 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q"] Sep 11 15:28:53.173640 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:28:53.173617 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bf98aa1_fa3d_4b06_b2b4_61a210957bcc.slice/crio-0456e77055e27cf61947c03e2fef9eea6d6c26321e297a5c4d90349216eb39a9 WatchSource:0}: Error finding container 0456e77055e27cf61947c03e2fef9eea6d6c26321e297a5c4d90349216eb39a9: Status 404 returned error can't find the container with id 0456e77055e27cf61947c03e2fef9eea6d6c26321e297a5c4d90349216eb39a9 Sep 11 15:28:53.437654 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:53.437574 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" event={"ID":"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc","Type":"ContainerStarted","Data":"f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d"} Sep 11 15:28:53.437654 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:53.437629 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" event={"ID":"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc","Type":"ContainerStarted","Data":"0456e77055e27cf61947c03e2fef9eea6d6c26321e297a5c4d90349216eb39a9"} Sep 11 15:28:53.454833 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:28:53.454758 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" podStartSLOduration=1.454744954 podStartE2EDuration="1.454744954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:28:53.452978504 +0000 UTC m=+1215.498303485" watchObservedRunningTime="2026-09-11 15:28:53.454744954 +0000 UTC m=+1215.500069979" Sep 11 15:29:02.898964 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:02.898922 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:29:02.898964 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:02.898979 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:29:02.911410 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:02.911386 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:29:03.511818 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:03.511792 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:29:20.209683 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:20.209648 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9"] Sep 11 15:29:20.213073 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:20.213056 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9" Sep 11 15:29:20.215664 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:20.215637 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:29:20.215816 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:20.215641 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:29:20.220885 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:20.220861 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9"] Sep 11 15:29:20.325260 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:20.325233 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1089ff7-cb5f-4ca8-a6dc-15359fd026a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9\" (UID: \"d1089ff7-cb5f-4ca8-a6dc-15359fd026a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9" Sep 11 15:29:20.426679 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:20.426655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1089ff7-cb5f-4ca8-a6dc-15359fd026a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9\" (UID: \"d1089ff7-cb5f-4ca8-a6dc-15359fd026a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9" Sep 11 15:29:20.427269 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:20.427252 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1089ff7-cb5f-4ca8-a6dc-15359fd026a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9\" (UID: \"d1089ff7-cb5f-4ca8-a6dc-15359fd026a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9" Sep 11 15:29:20.526184 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:20.526120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9" Sep 11 15:29:20.795351 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:20.795326 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9"] Sep 11 15:29:21.604874 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:21.604836 2573 generic.go:356] "Generic (PLEG): container finished" podID="d1089ff7-cb5f-4ca8-a6dc-15359fd026a3" containerID="4f87dc106edb793049a23315a18312eb631d2e99299535aef6f6dc1e24278bdc" exitCode=0 Sep 11 15:29:21.605166 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:21.604933 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9" event={"ID":"d1089ff7-cb5f-4ca8-a6dc-15359fd026a3","Type":"ContainerDied","Data":"4f87dc106edb793049a23315a18312eb631d2e99299535aef6f6dc1e24278bdc"} Sep 11 15:29:21.605166 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:21.604986 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9" event={"ID":"d1089ff7-cb5f-4ca8-a6dc-15359fd026a3","Type":"ContainerStarted","Data":"7cf361b051c0c1ae3c2d271eb4678e110a91179ed1e7fe39b2bbbf2624ddc51e"} Sep 11 15:29:22.744520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:22.744497 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9" Sep 11 15:29:22.850830 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:22.850793 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1089ff7-cb5f-4ca8-a6dc-15359fd026a3-maas-api-service-ca\") pod \"d1089ff7-cb5f-4ca8-a6dc-15359fd026a3\" (UID: \"d1089ff7-cb5f-4ca8-a6dc-15359fd026a3\") " Sep 11 15:29:22.851201 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:22.851180 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1089ff7-cb5f-4ca8-a6dc-15359fd026a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1089ff7-cb5f-4ca8-a6dc-15359fd026a3" (UID: "d1089ff7-cb5f-4ca8-a6dc-15359fd026a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:29:22.952222 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:22.952163 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1089ff7-cb5f-4ca8-a6dc-15359fd026a3-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:29:23.617367 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:23.617339 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9" Sep 11 15:29:23.617520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:23.617333 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9" event={"ID":"d1089ff7-cb5f-4ca8-a6dc-15359fd026a3","Type":"ContainerDied","Data":"7cf361b051c0c1ae3c2d271eb4678e110a91179ed1e7fe39b2bbbf2624ddc51e"} Sep 11 15:29:23.617520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:23.617461 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cf361b051c0c1ae3c2d271eb4678e110a91179ed1e7fe39b2bbbf2624ddc51e" Sep 11 15:29:35.950881 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:35.950799 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9"] Sep 11 15:29:35.953621 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:35.953596 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dcdcc9ba-r7lg9"] Sep 11 15:29:36.493657 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:36.493628 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1089ff7-cb5f-4ca8-a6dc-15359fd026a3" path="/var/lib/kubelet/pods/d1089ff7-cb5f-4ca8-a6dc-15359fd026a3/volumes" Sep 11 15:29:38.634912 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:38.634882 2573 scope.go:117] "RemoveContainer" containerID="6ea39e8d21455a967be079521e0c150cc656ca3bb28e6e0ab06e3bd3aa480a3f" Sep 11 15:29:38.643822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:38.643801 2573 scope.go:117] "RemoveContainer" containerID="d7819f9d2a5a3e021942a5c63d26055494b51893ccbec851913cb4eaa6273f15" Sep 11 15:29:38.651544 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:38.651526 2573 scope.go:117] "RemoveContainer" containerID="065f27410ab6c6282ebd4f148ac1bd6c3da8e141e3fe8246f8fcdb30ad148aa9" Sep 11 15:29:44.341181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:44.341150 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q"] Sep 11 15:29:59.360800 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.360713 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" podUID="0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" containerName="main" containerID="cri-o://f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d" gracePeriod=45 Sep 11 15:29:59.617640 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.617583 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:29:59.660273 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660243 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-home\") pod \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " Sep 11 15:29:59.660413 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660306 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls9pk\" (UniqueName: \"kubernetes.io/projected/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-kube-api-access-ls9pk\") pod \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " Sep 11 15:29:59.660413 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660333 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-model-cache\") pod \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " Sep 11 15:29:59.660413 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660364 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tls-certs\") pod \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " Sep 11 15:29:59.660594 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660438 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-dshm\") pod \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " Sep 11 15:29:59.660594 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660479 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tmp-dir\") pod \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\" (UID: \"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc\") " Sep 11 15:29:59.660700 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660611 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" (UID: "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:29:59.660755 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660737 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-home" (OuterVolumeSpecName: "home") pod "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" (UID: "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:29:59.660932 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660907 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" (UID: "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:29:59.661005 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660943 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-home\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:29:59.661005 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.660964 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:29:59.663081 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.663054 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" (UID: "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:29:59.663204 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.663181 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-kube-api-access-ls9pk" (OuterVolumeSpecName: "kube-api-access-ls9pk") pod "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" (UID: "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc"). InnerVolumeSpecName "kube-api-access-ls9pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:29:59.663251 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.663228 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-dshm" (OuterVolumeSpecName: "dshm") pod "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" (UID: "0bf98aa1-fa3d-4b06-b2b4-61a210957bcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:29:59.761500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.761476 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls9pk\" (UniqueName: \"kubernetes.io/projected/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-kube-api-access-ls9pk\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:29:59.761500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.761499 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tls-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:29:59.761646 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.761510 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-dshm\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:29:59.761646 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.761537 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc-tmp-dir\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:29:59.806409 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.806387 2573 generic.go:356] "Generic (PLEG): container finished" podID="0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" containerID="f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d" exitCode=0 Sep 11 15:29:59.806520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.806436 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" Sep 11 15:29:59.806520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.806482 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" event={"ID":"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc","Type":"ContainerDied","Data":"f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d"} Sep 11 15:29:59.806600 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.806524 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q" event={"ID":"0bf98aa1-fa3d-4b06-b2b4-61a210957bcc","Type":"ContainerDied","Data":"0456e77055e27cf61947c03e2fef9eea6d6c26321e297a5c4d90349216eb39a9"} Sep 11 15:29:59.806600 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.806544 2573 scope.go:117] "RemoveContainer" containerID="f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d" Sep 11 15:29:59.815865 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.815850 2573 scope.go:117] "RemoveContainer" containerID="f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d" Sep 11 15:29:59.816134 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:29:59.816116 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d\": container with ID starting with f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d not found: ID does not exist" containerID="f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d" Sep 11 15:29:59.816182 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.816142 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d"} err="failed to get container status \"f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d\": rpc error: code = NotFound desc = could not find container \"f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d\": container with ID starting with f073f0f784d246b5bd02a6b4e9f801e20d78963f1c2633aa59cfdabd876cdf8d not found: ID does not exist" Sep 11 15:29:59.828297 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.828276 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q"] Sep 11 15:29:59.830928 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:29:59.830910 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-dcdcc9ba/e2e-lifecycle-model-dcdcc9ba-kserve-cf47f8887-8cs6q"] Sep 11 15:30:00.141721 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.141694 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28"] Sep 11 15:30:00.142136 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.142122 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" containerName="main" Sep 11 15:30:00.142199 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.142137 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" containerName="main" Sep 11 15:30:00.142199 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.142148 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1089ff7-cb5f-4ca8-a6dc-15359fd026a3" containerName="revoke-keys" Sep 11 15:30:00.142199 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.142154 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1089ff7-cb5f-4ca8-a6dc-15359fd026a3" containerName="revoke-keys" Sep 11 15:30:00.142308 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.142211 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" containerName="main" Sep 11 15:30:00.142308 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.142218 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1089ff7-cb5f-4ca8-a6dc-15359fd026a3" containerName="revoke-keys" Sep 11 15:30:00.146997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.146978 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28" Sep 11 15:30:00.148046 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.148029 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn"] Sep 11 15:30:00.149299 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.149249 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:30:00.151488 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.151469 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b"] Sep 11 15:30:00.151619 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.151599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn" Sep 11 15:30:00.155437 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.155275 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28"] Sep 11 15:30:00.155437 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.155369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b" Sep 11 15:30:00.158743 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.158689 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn"] Sep 11 15:30:00.161386 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.161361 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b"] Sep 11 15:30:00.161676 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.161656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28" Sep 11 15:30:00.174883 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.174864 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn" Sep 11 15:30:00.183033 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.183009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b" Sep 11 15:30:00.443944 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.443920 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28"] Sep 11 15:30:00.446042 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:30:00.446010 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc274876b_e3df_4fe3_adcd_3654dd677bb4.slice/crio-5b5e9cdc3ca0f8e6ccf8f50900ca39526de1d0ef56ca426139458eead852a65f WatchSource:0}: Error finding container 5b5e9cdc3ca0f8e6ccf8f50900ca39526de1d0ef56ca426139458eead852a65f: Status 404 returned error can't find the container with id 5b5e9cdc3ca0f8e6ccf8f50900ca39526de1d0ef56ca426139458eead852a65f Sep 11 15:30:00.470643 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.470622 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn"] Sep 11 15:30:00.471633 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:30:00.471613 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf3f787_2f12_404b_b832_530a4521a9f3.slice/crio-7cbf38a33fee86a5f07366f25095be3b85659e97ae9cf4425b3f2854b6173e92 WatchSource:0}: Error finding container 7cbf38a33fee86a5f07366f25095be3b85659e97ae9cf4425b3f2854b6173e92: Status 404 returned error can't find the container with id 7cbf38a33fee86a5f07366f25095be3b85659e97ae9cf4425b3f2854b6173e92 Sep 11 15:30:00.480964 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.480939 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b"] Sep 11 15:30:00.492885 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:30:00.492857 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2724c5_49b0_4d33_bd44_9ef32af6fd9c.slice/crio-e7a59367a765e0c8f24973672dfaaa4d9117d509a62a79a0589449f7eb02fd65 WatchSource:0}: Error finding container e7a59367a765e0c8f24973672dfaaa4d9117d509a62a79a0589449f7eb02fd65: Status 404 returned error can't find the container with id e7a59367a765e0c8f24973672dfaaa4d9117d509a62a79a0589449f7eb02fd65 Sep 11 15:30:00.495466 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.495442 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bf98aa1-fa3d-4b06-b2b4-61a210957bcc" path="/var/lib/kubelet/pods/0bf98aa1-fa3d-4b06-b2b4-61a210957bcc/volumes" Sep 11 15:30:00.814739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.814710 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn" event={"ID":"bbf3f787-2f12-404b-b832-530a4521a9f3","Type":"ContainerStarted","Data":"1eb3d0c67384b4f91ed645ad15a1e09d038c9570d94f30d4a4fa9b086d7d553d"} Sep 11 15:30:00.814866 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.814767 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn" event={"ID":"bbf3f787-2f12-404b-b832-530a4521a9f3","Type":"ContainerStarted","Data":"7cbf38a33fee86a5f07366f25095be3b85659e97ae9cf4425b3f2854b6173e92"} Sep 11 15:30:00.816309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.816282 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b" event={"ID":"7e2724c5-49b0-4d33-bd44-9ef32af6fd9c","Type":"ContainerStarted","Data":"e7a59367a765e0c8f24973672dfaaa4d9117d509a62a79a0589449f7eb02fd65"} Sep 11 15:30:00.818001 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.817980 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28" event={"ID":"c274876b-e3df-4fe3-adcd-3654dd677bb4","Type":"ContainerStarted","Data":"6a47bd0ea340ee6ddd1c194970eb783757a20b6f741133735bf6ffddb71dd93b"} Sep 11 15:30:00.818088 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.818027 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28" event={"ID":"c274876b-e3df-4fe3-adcd-3654dd677bb4","Type":"ContainerStarted","Data":"5b5e9cdc3ca0f8e6ccf8f50900ca39526de1d0ef56ca426139458eead852a65f"} Sep 11 15:30:00.835365 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.835313 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn" podStartSLOduration=0.835295745 podStartE2EDuration="835.295745ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:30:00.83493684 +0000 UTC m=+1282.880261821" watchObservedRunningTime="2026-09-11 15:30:00.835295745 +0000 UTC m=+1282.880620727" Sep 11 15:30:00.872391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.872269 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28" podStartSLOduration=0.872253401 podStartE2EDuration="872.253401ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:30:00.87200147 +0000 UTC m=+1282.917326501" watchObservedRunningTime="2026-09-11 15:30:00.872253401 +0000 UTC m=+1282.917578383" Sep 11 15:30:00.872944 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:00.872910 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b" podStartSLOduration=0.872893655 podStartE2EDuration="872.893655ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:30:00.854857982 +0000 UTC m=+1282.900182964" watchObservedRunningTime="2026-09-11 15:30:00.872893655 +0000 UTC m=+1282.918218635" Sep 11 15:30:01.825440 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:01.825399 2573 generic.go:356] "Generic (PLEG): container finished" podID="c274876b-e3df-4fe3-adcd-3654dd677bb4" containerID="6a47bd0ea340ee6ddd1c194970eb783757a20b6f741133735bf6ffddb71dd93b" exitCode=0 Sep 11 15:30:01.825886 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:01.825485 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28" event={"ID":"c274876b-e3df-4fe3-adcd-3654dd677bb4","Type":"ContainerDied","Data":"6a47bd0ea340ee6ddd1c194970eb783757a20b6f741133735bf6ffddb71dd93b"} Sep 11 15:30:01.827142 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:01.827118 2573 generic.go:356] "Generic (PLEG): container finished" podID="bbf3f787-2f12-404b-b832-530a4521a9f3" containerID="1eb3d0c67384b4f91ed645ad15a1e09d038c9570d94f30d4a4fa9b086d7d553d" exitCode=0 Sep 11 15:30:01.827266 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:01.827211 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn" event={"ID":"bbf3f787-2f12-404b-b832-530a4521a9f3","Type":"ContainerDied","Data":"1eb3d0c67384b4f91ed645ad15a1e09d038c9570d94f30d4a4fa9b086d7d553d"} Sep 11 15:30:01.828732 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:01.828707 2573 generic.go:356] "Generic (PLEG): container finished" podID="7e2724c5-49b0-4d33-bd44-9ef32af6fd9c" containerID="771c6a40f4b69d494e9684fdd10a18fc638887d64b72302b19b0875ba624e52b" exitCode=0 Sep 11 15:30:01.828834 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:01.828751 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b" event={"ID":"7e2724c5-49b0-4d33-bd44-9ef32af6fd9c","Type":"ContainerDied","Data":"771c6a40f4b69d494e9684fdd10a18fc638887d64b72302b19b0875ba624e52b"} Sep 11 15:30:02.982331 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:02.982273 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28" Sep 11 15:30:03.015503 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.015415 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b" Sep 11 15:30:03.047089 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.047069 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn" Sep 11 15:30:03.843502 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.843473 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28" Sep 11 15:30:03.843679 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.843508 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28" event={"ID":"c274876b-e3df-4fe3-adcd-3654dd677bb4","Type":"ContainerDied","Data":"5b5e9cdc3ca0f8e6ccf8f50900ca39526de1d0ef56ca426139458eead852a65f"} Sep 11 15:30:03.843679 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.843559 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b5e9cdc3ca0f8e6ccf8f50900ca39526de1d0ef56ca426139458eead852a65f" Sep 11 15:30:03.844955 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.844936 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn" Sep 11 15:30:03.845090 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.844972 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn" event={"ID":"bbf3f787-2f12-404b-b832-530a4521a9f3","Type":"ContainerDied","Data":"7cbf38a33fee86a5f07366f25095be3b85659e97ae9cf4425b3f2854b6173e92"} Sep 11 15:30:03.845090 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.845015 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cbf38a33fee86a5f07366f25095be3b85659e97ae9cf4425b3f2854b6173e92" Sep 11 15:30:03.846360 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.846343 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b" Sep 11 15:30:03.846457 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.846340 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b" event={"ID":"7e2724c5-49b0-4d33-bd44-9ef32af6fd9c","Type":"ContainerDied","Data":"e7a59367a765e0c8f24973672dfaaa4d9117d509a62a79a0589449f7eb02fd65"} Sep 11 15:30:03.846502 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:03.846477 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7a59367a765e0c8f24973672dfaaa4d9117d509a62a79a0589449f7eb02fd65" Sep 11 15:30:24.193104 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.193072 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7340-650215"] Sep 11 15:30:24.193500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.193458 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf3f787-2f12-404b-b832-530a4521a9f3" containerName="cleanup" Sep 11 15:30:24.193500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.193469 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf3f787-2f12-404b-b832-530a4521a9f3" containerName="cleanup" Sep 11 15:30:24.193500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.193475 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c274876b-e3df-4fe3-adcd-3654dd677bb4" containerName="cleanup" Sep 11 15:30:24.193500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.193480 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c274876b-e3df-4fe3-adcd-3654dd677bb4" containerName="cleanup" Sep 11 15:30:24.193872 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.193504 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e2724c5-49b0-4d33-bd44-9ef32af6fd9c" containerName="cleanup" Sep 11 15:30:24.193872 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.193511 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e2724c5-49b0-4d33-bd44-9ef32af6fd9c" containerName="cleanup" Sep 11 15:30:24.193872 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.193571 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c274876b-e3df-4fe3-adcd-3654dd677bb4" containerName="cleanup" Sep 11 15:30:24.193872 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.193579 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbf3f787-2f12-404b-b832-530a4521a9f3" containerName="cleanup" Sep 11 15:30:24.193872 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.193585 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e2724c5-49b0-4d33-bd44-9ef32af6fd9c" containerName="cleanup" Sep 11 15:30:24.196736 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.196717 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-650215" Sep 11 15:30:24.199117 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.199093 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6mbx9\"" Sep 11 15:30:24.202813 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.200704 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7340-650215"] Sep 11 15:30:24.275261 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.275230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v44gq\" (UniqueName: \"kubernetes.io/projected/63f87e3f-a72a-4725-a15e-ad6e7e43a149-kube-api-access-v44gq\") pod \"test-curl-7340-650215\" (UID: \"63f87e3f-a72a-4725-a15e-ad6e7e43a149\") " pod="opendatahub/test-curl-7340-650215" Sep 11 15:30:24.376232 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.376204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v44gq\" (UniqueName: \"kubernetes.io/projected/63f87e3f-a72a-4725-a15e-ad6e7e43a149-kube-api-access-v44gq\") pod \"test-curl-7340-650215\" (UID: \"63f87e3f-a72a-4725-a15e-ad6e7e43a149\") " pod="opendatahub/test-curl-7340-650215" Sep 11 15:30:24.384504 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.384484 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v44gq\" (UniqueName: \"kubernetes.io/projected/63f87e3f-a72a-4725-a15e-ad6e7e43a149-kube-api-access-v44gq\") pod \"test-curl-7340-650215\" (UID: \"63f87e3f-a72a-4725-a15e-ad6e7e43a149\") " pod="opendatahub/test-curl-7340-650215" Sep 11 15:30:24.513092 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.513037 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-650215" Sep 11 15:30:24.780949 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.780923 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7340-650215"] Sep 11 15:30:24.782383 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:30:24.782351 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63f87e3f_a72a_4725_a15e_ad6e7e43a149.slice/crio-e440765a4b5e54a988168f1f8d038117041fdc5ee883a0a733d06564e8854280 WatchSource:0}: Error finding container e440765a4b5e54a988168f1f8d038117041fdc5ee883a0a733d06564e8854280: Status 404 returned error can't find the container with id e440765a4b5e54a988168f1f8d038117041fdc5ee883a0a733d06564e8854280 Sep 11 15:30:24.959216 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.959182 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7340-650215" event={"ID":"63f87e3f-a72a-4725-a15e-ad6e7e43a149","Type":"ContainerStarted","Data":"c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941"} Sep 11 15:30:24.959335 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.959231 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7340-650215" event={"ID":"63f87e3f-a72a-4725-a15e-ad6e7e43a149","Type":"ContainerStarted","Data":"e440765a4b5e54a988168f1f8d038117041fdc5ee883a0a733d06564e8854280"} Sep 11 15:30:24.974066 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:24.974021 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7340-650215" podStartSLOduration=0.974007596 podStartE2EDuration="974.007596ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:30:24.973808321 +0000 UTC m=+1307.019133298" watchObservedRunningTime="2026-09-11 15:30:24.974007596 +0000 UTC m=+1307.019332576" Sep 11 15:30:25.542912 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:25.542831 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7340-650215"] Sep 11 15:30:25.547937 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:25.547907 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7340-650215"] Sep 11 15:30:25.964905 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:30:25.964880 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7340-650215" container="test-curl-7340-650215" Sep 11 15:30:26.975160 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:26.975117 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7340-650215" podUID="63f87e3f-a72a-4725-a15e-ad6e7e43a149" containerName="test-curl-7340-650215" containerID="cri-o://c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941" gracePeriod=2 Sep 11 15:30:26.976130 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:26.976107 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r"] Sep 11 15:30:28.078466 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:28.078432 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6"] Sep 11 15:30:29.159060 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:29.159035 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-650215" Sep 11 15:30:29.224254 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:29.224232 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v44gq\" (UniqueName: \"kubernetes.io/projected/63f87e3f-a72a-4725-a15e-ad6e7e43a149-kube-api-access-v44gq\") pod \"63f87e3f-a72a-4725-a15e-ad6e7e43a149\" (UID: \"63f87e3f-a72a-4725-a15e-ad6e7e43a149\") " Sep 11 15:30:29.226324 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:29.226294 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f87e3f-a72a-4725-a15e-ad6e7e43a149-kube-api-access-v44gq" (OuterVolumeSpecName: "kube-api-access-v44gq") pod "63f87e3f-a72a-4725-a15e-ad6e7e43a149" (UID: "63f87e3f-a72a-4725-a15e-ad6e7e43a149"). InnerVolumeSpecName "kube-api-access-v44gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:30:29.325167 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:29.325140 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v44gq\" (UniqueName: \"kubernetes.io/projected/63f87e3f-a72a-4725-a15e-ad6e7e43a149-kube-api-access-v44gq\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:29.992713 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:29.992678 2573 generic.go:356] "Generic (PLEG): container finished" podID="63f87e3f-a72a-4725-a15e-ad6e7e43a149" containerID="c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941" exitCode=137 Sep 11 15:30:29.992931 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:29.992718 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-650215" Sep 11 15:30:29.992931 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:29.992754 2573 scope.go:117] "RemoveContainer" containerID="c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941" Sep 11 15:30:30.002009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:30.001986 2573 scope.go:117] "RemoveContainer" containerID="c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941" Sep 11 15:30:30.002310 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:30:30.002289 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941\": container with ID starting with c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941 not found: ID does not exist" containerID="c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941" Sep 11 15:30:30.002357 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:30.002320 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941"} err="failed to get container status \"c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941\": rpc error: code = NotFound desc = could not find container \"c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941\": container with ID starting with c8236996d2b36c87e3b518c22ded535110e836749515f62ca72ca8cc15143941 not found: ID does not exist" Sep 11 15:30:30.003228 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:30:30.003206 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7340-650215\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="63f87e3f-a72a-4725-a15e-ad6e7e43a149" pod="opendatahub/test-curl-7340-650215" Sep 11 15:30:30.493257 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:30.493224 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f87e3f-a72a-4725-a15e-ad6e7e43a149" path="/var/lib/kubelet/pods/63f87e3f-a72a-4725-a15e-ad6e7e43a149/volumes" Sep 11 15:30:38.680475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:38.680444 2573 scope.go:117] "RemoveContainer" containerID="3ccab57c688e33a0c67af8090f968118bc9ef8fee789ea9e0cf517300e5cf76b" Sep 11 15:30:41.996138 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:41.996095 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" podUID="fb64fc4d-4ee9-4025-9996-54b3051c7e5c" containerName="main" containerID="cri-o://699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af" gracePeriod=45 Sep 11 15:30:42.255808 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.255738 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:30:42.341789 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.341744 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-model-cache\") pod \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " Sep 11 15:30:42.341930 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.341813 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tls-certs\") pod \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " Sep 11 15:30:42.341930 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.341837 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh8ld\" (UniqueName: \"kubernetes.io/projected/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-kube-api-access-zh8ld\") pod \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " Sep 11 15:30:42.341930 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.341873 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-home\") pod \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " Sep 11 15:30:42.342078 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.341949 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-dshm\") pod \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " Sep 11 15:30:42.342078 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.341986 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tmp-dir\") pod \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\" (UID: \"fb64fc4d-4ee9-4025-9996-54b3051c7e5c\") " Sep 11 15:30:42.342078 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.341985 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb64fc4d-4ee9-4025-9996-54b3051c7e5c" (UID: "fb64fc4d-4ee9-4025-9996-54b3051c7e5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:42.342276 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.342254 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-home" (OuterVolumeSpecName: "home") pod "fb64fc4d-4ee9-4025-9996-54b3051c7e5c" (UID: "fb64fc4d-4ee9-4025-9996-54b3051c7e5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:42.342343 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.342279 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb64fc4d-4ee9-4025-9996-54b3051c7e5c" (UID: "fb64fc4d-4ee9-4025-9996-54b3051c7e5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:42.342408 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.342376 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tmp-dir\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:42.342408 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.342396 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:42.342486 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.342410 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-home\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:42.344215 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.344190 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-dshm" (OuterVolumeSpecName: "dshm") pod "fb64fc4d-4ee9-4025-9996-54b3051c7e5c" (UID: "fb64fc4d-4ee9-4025-9996-54b3051c7e5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:42.344337 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.344293 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-kube-api-access-zh8ld" (OuterVolumeSpecName: "kube-api-access-zh8ld") pod "fb64fc4d-4ee9-4025-9996-54b3051c7e5c" (UID: "fb64fc4d-4ee9-4025-9996-54b3051c7e5c"). InnerVolumeSpecName "kube-api-access-zh8ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:30:42.344387 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.344359 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb64fc4d-4ee9-4025-9996-54b3051c7e5c" (UID: "fb64fc4d-4ee9-4025-9996-54b3051c7e5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:30:42.443022 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.442994 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-tls-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:42.443022 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.443018 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8ld\" (UniqueName: \"kubernetes.io/projected/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-kube-api-access-zh8ld\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:42.443156 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:42.443027 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb64fc4d-4ee9-4025-9996-54b3051c7e5c-dshm\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:43.061745 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.061715 2573 generic.go:356] "Generic (PLEG): container finished" podID="fb64fc4d-4ee9-4025-9996-54b3051c7e5c" containerID="699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af" exitCode=0 Sep 11 15:30:43.062137 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.061759 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" Sep 11 15:30:43.062137 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.061813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" event={"ID":"fb64fc4d-4ee9-4025-9996-54b3051c7e5c","Type":"ContainerDied","Data":"699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af"} Sep 11 15:30:43.062137 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.061858 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r" event={"ID":"fb64fc4d-4ee9-4025-9996-54b3051c7e5c","Type":"ContainerDied","Data":"f838b97179c9f772410c6161c31b75562944713318bdb85184b4b413a6b1658f"} Sep 11 15:30:43.062137 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.061873 2573 scope.go:117] "RemoveContainer" containerID="699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af" Sep 11 15:30:43.071253 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.071234 2573 scope.go:117] "RemoveContainer" containerID="699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af" Sep 11 15:30:43.071537 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:30:43.071517 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af\": container with ID starting with 699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af not found: ID does not exist" containerID="699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af" Sep 11 15:30:43.071602 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.071546 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af"} err="failed to get container status \"699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af\": rpc error: code = NotFound desc = could not find container \"699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af\": container with ID starting with 699b4a63c0f75ec94c3ff2fc45414319a02ae14e0789973d0e657166b69858af not found: ID does not exist" Sep 11 15:30:43.077485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.077460 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r"] Sep 11 15:30:43.079410 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.079394 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/auth-test-model-397e40-kserve-f6b97d499-6t78r"] Sep 11 15:30:43.097898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.097873 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" podUID="dde3d533-8dba-46f1-9281-a00f9586dcca" containerName="main" containerID="cri-o://045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8" gracePeriod=45 Sep 11 15:30:43.342363 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.342340 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:30:43.450083 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450061 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde3d533-8dba-46f1-9281-a00f9586dcca-tls-certs\") pod \"dde3d533-8dba-46f1-9281-a00f9586dcca\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " Sep 11 15:30:43.450216 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450140 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-dshm\") pod \"dde3d533-8dba-46f1-9281-a00f9586dcca\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " Sep 11 15:30:43.450216 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450165 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-home\") pod \"dde3d533-8dba-46f1-9281-a00f9586dcca\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " Sep 11 15:30:43.450216 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450195 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-tmp-dir\") pod \"dde3d533-8dba-46f1-9281-a00f9586dcca\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " Sep 11 15:30:43.450391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450228 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-model-cache\") pod \"dde3d533-8dba-46f1-9281-a00f9586dcca\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " Sep 11 15:30:43.450391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450278 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r47dc\" (UniqueName: \"kubernetes.io/projected/dde3d533-8dba-46f1-9281-a00f9586dcca-kube-api-access-r47dc\") pod \"dde3d533-8dba-46f1-9281-a00f9586dcca\" (UID: \"dde3d533-8dba-46f1-9281-a00f9586dcca\") " Sep 11 15:30:43.450499 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450450 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-home" (OuterVolumeSpecName: "home") pod "dde3d533-8dba-46f1-9281-a00f9586dcca" (UID: "dde3d533-8dba-46f1-9281-a00f9586dcca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:43.450499 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450461 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dde3d533-8dba-46f1-9281-a00f9586dcca" (UID: "dde3d533-8dba-46f1-9281-a00f9586dcca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:43.450602 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450565 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-model-cache" (OuterVolumeSpecName: "model-cache") pod "dde3d533-8dba-46f1-9281-a00f9586dcca" (UID: "dde3d533-8dba-46f1-9281-a00f9586dcca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:43.450766 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450713 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-home\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:43.450766 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450739 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-tmp-dir\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:43.450937 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.450754 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:43.452517 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.452492 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-dshm" (OuterVolumeSpecName: "dshm") pod "dde3d533-8dba-46f1-9281-a00f9586dcca" (UID: "dde3d533-8dba-46f1-9281-a00f9586dcca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:43.452517 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.452492 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde3d533-8dba-46f1-9281-a00f9586dcca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dde3d533-8dba-46f1-9281-a00f9586dcca" (UID: "dde3d533-8dba-46f1-9281-a00f9586dcca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:30:43.452636 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.452530 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde3d533-8dba-46f1-9281-a00f9586dcca-kube-api-access-r47dc" (OuterVolumeSpecName: "kube-api-access-r47dc") pod "dde3d533-8dba-46f1-9281-a00f9586dcca" (UID: "dde3d533-8dba-46f1-9281-a00f9586dcca"). InnerVolumeSpecName "kube-api-access-r47dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:30:43.552872 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.552849 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde3d533-8dba-46f1-9281-a00f9586dcca-tls-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:43.552975 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.552876 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde3d533-8dba-46f1-9281-a00f9586dcca-dshm\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:43.552975 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:43.552891 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r47dc\" (UniqueName: \"kubernetes.io/projected/dde3d533-8dba-46f1-9281-a00f9586dcca-kube-api-access-r47dc\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:30:44.069928 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.069895 2573 generic.go:356] "Generic (PLEG): container finished" podID="dde3d533-8dba-46f1-9281-a00f9586dcca" containerID="045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8" exitCode=0 Sep 11 15:30:44.070383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.069942 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" Sep 11 15:30:44.070383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.069951 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" event={"ID":"dde3d533-8dba-46f1-9281-a00f9586dcca","Type":"ContainerDied","Data":"045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8"} Sep 11 15:30:44.070383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.069994 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6" event={"ID":"dde3d533-8dba-46f1-9281-a00f9586dcca","Type":"ContainerDied","Data":"2fc95fa3f45b7e52ba14dc7931e8f4194f7abc0626f04283ae585b1bf6779f63"} Sep 11 15:30:44.070383 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.070019 2573 scope.go:117] "RemoveContainer" containerID="045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8" Sep 11 15:30:44.080002 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.079973 2573 scope.go:117] "RemoveContainer" containerID="045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8" Sep 11 15:30:44.080313 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:30:44.080293 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8\": container with ID starting with 045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8 not found: ID does not exist" containerID="045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8" Sep 11 15:30:44.080385 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.080322 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8"} err="failed to get container status \"045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8\": rpc error: code = NotFound desc = could not find container \"045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8\": container with ID starting with 045178dbfc0cd1096d58d981a2a01018c767d570e7aad9d565ab992109758fe8 not found: ID does not exist" Sep 11 15:30:44.093321 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.093297 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6"] Sep 11 15:30:44.097161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.097134 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-abba31/auth-test-model-abba31-kserve-848fbdbd59-9scs6"] Sep 11 15:30:44.493651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.493619 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dde3d533-8dba-46f1-9281-a00f9586dcca" path="/var/lib/kubelet/pods/dde3d533-8dba-46f1-9281-a00f9586dcca/volumes" Sep 11 15:30:44.494079 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.494059 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb64fc4d-4ee9-4025-9996-54b3051c7e5c" path="/var/lib/kubelet/pods/fb64fc4d-4ee9-4025-9996-54b3051c7e5c/volumes" Sep 11 15:30:44.996140 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.996110 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2"] Sep 11 15:30:44.996546 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.996532 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f87e3f-a72a-4725-a15e-ad6e7e43a149" containerName="test-curl-7340-650215" Sep 11 15:30:44.996598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.996549 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f87e3f-a72a-4725-a15e-ad6e7e43a149" containerName="test-curl-7340-650215" Sep 11 15:30:44.996598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.996576 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb64fc4d-4ee9-4025-9996-54b3051c7e5c" containerName="main" Sep 11 15:30:44.996598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.996582 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb64fc4d-4ee9-4025-9996-54b3051c7e5c" containerName="main" Sep 11 15:30:44.996598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.996588 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dde3d533-8dba-46f1-9281-a00f9586dcca" containerName="main" Sep 11 15:30:44.996598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.996593 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dde3d533-8dba-46f1-9281-a00f9586dcca" containerName="main" Sep 11 15:30:44.996747 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.996667 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dde3d533-8dba-46f1-9281-a00f9586dcca" containerName="main" Sep 11 15:30:44.996747 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.996676 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f87e3f-a72a-4725-a15e-ad6e7e43a149" containerName="test-curl-7340-650215" Sep 11 15:30:44.996747 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:44.996683 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb64fc4d-4ee9-4025-9996-54b3051c7e5c" containerName="main" Sep 11 15:30:45.000898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.000880 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.004005 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.003982 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-397e40\"/\"default-dockercfg-bjs5g\"" Sep 11 15:30:45.004005 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.004001 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-397e40\"/\"openshift-service-ca.crt\"" Sep 11 15:30:45.004181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.003984 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-397e40\"/\"sub-test-model-397e40-kserve-self-signed-certs\"" Sep 11 15:30:45.004181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.003990 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-397e40\"/\"kube-root-ca.crt\"" Sep 11 15:30:45.011033 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.011013 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2"] Sep 11 15:30:45.066946 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.066917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-dshm\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.066946 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.066947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0eb8228-53a8-4200-b725-2dad54755b57-tls-certs\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.067082 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.066969 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-tmp-dir\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.067082 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.067029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2prhp\" (UniqueName: \"kubernetes.io/projected/f0eb8228-53a8-4200-b725-2dad54755b57-kube-api-access-2prhp\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.067082 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.067076 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-home\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.067180 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.067124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-model-cache\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.168190 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.168165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2prhp\" (UniqueName: \"kubernetes.io/projected/f0eb8228-53a8-4200-b725-2dad54755b57-kube-api-access-2prhp\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.168516 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.168207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-home\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.168516 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.168231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-model-cache\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.168516 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.168285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-dshm\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.168516 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.168300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0eb8228-53a8-4200-b725-2dad54755b57-tls-certs\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.168516 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.168327 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-tmp-dir\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.168689 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.168584 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-home\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.168689 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.168648 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-model-cache\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.168760 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.168732 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-tmp-dir\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.170831 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.170802 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-dshm\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.171077 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.171062 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0eb8228-53a8-4200-b725-2dad54755b57-tls-certs\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.176583 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.176562 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2prhp\" (UniqueName: \"kubernetes.io/projected/f0eb8228-53a8-4200-b725-2dad54755b57-kube-api-access-2prhp\") pod \"sub-test-model-397e40-kserve-84df859fcf-kgrq2\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.324948 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.324922 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:45.595130 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:45.595105 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2"] Sep 11 15:30:45.596439 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:30:45.596414 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0eb8228_53a8_4200_b725_2dad54755b57.slice/crio-db5e784518dcfaa8d86ba70a8fb0a068f76a38290174a7e82fe0d2b5db98570c WatchSource:0}: Error finding container db5e784518dcfaa8d86ba70a8fb0a068f76a38290174a7e82fe0d2b5db98570c: Status 404 returned error can't find the container with id db5e784518dcfaa8d86ba70a8fb0a068f76a38290174a7e82fe0d2b5db98570c Sep 11 15:30:46.084264 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:46.084228 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" event={"ID":"f0eb8228-53a8-4200-b725-2dad54755b57","Type":"ContainerStarted","Data":"ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000"} Sep 11 15:30:46.084264 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:46.084268 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" event={"ID":"f0eb8228-53a8-4200-b725-2dad54755b57","Type":"ContainerStarted","Data":"db5e784518dcfaa8d86ba70a8fb0a068f76a38290174a7e82fe0d2b5db98570c"} Sep 11 15:30:46.102059 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:46.102011 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" podStartSLOduration=2.101996118 podStartE2EDuration="2.101996118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:30:46.099199918 +0000 UTC m=+1328.144524897" watchObservedRunningTime="2026-09-11 15:30:46.101996118 +0000 UTC m=+1328.147321097" Sep 11 15:30:55.325758 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:55.325727 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:55.326289 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:55.325805 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:55.338287 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:55.338259 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:30:56.156751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:30:56.156719 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:31:28.748573 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.748541 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq"] Sep 11 15:31:28.752400 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.752378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:31:28.755068 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.755047 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-808212\"" Sep 11 15:31:28.755188 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.755051 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-808212-dockercfg-k8q2r\"" Sep 11 15:31:28.765540 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.765517 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq"] Sep 11 15:31:28.823721 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.823691 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmtq2\" (UniqueName: \"kubernetes.io/projected/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-kube-api-access-fmtq2\") pod \"payload-processing-e2e-api-a-808212-79874d56df-gdhwq\" (UID: \"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:31:28.823855 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.823791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-808212-79874d56df-gdhwq\" (UID: \"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:31:28.924692 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.924662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-808212-79874d56df-gdhwq\" (UID: \"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:31:28.924836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.924744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmtq2\" (UniqueName: \"kubernetes.io/projected/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-kube-api-access-fmtq2\") pod \"payload-processing-e2e-api-a-808212-79874d56df-gdhwq\" (UID: \"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:31:28.925330 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.925312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-808212-79874d56df-gdhwq\" (UID: \"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:31:28.933650 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:28.933632 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmtq2\" (UniqueName: \"kubernetes.io/projected/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-kube-api-access-fmtq2\") pod \"payload-processing-e2e-api-a-808212-79874d56df-gdhwq\" (UID: \"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc\") " pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:31:29.063464 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:29.063402 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:31:29.338254 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:29.338231 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq"] Sep 11 15:31:29.340089 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:31:29.340062 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9be4f5_d6aa_4a94_b6ff_c20144d0f5bc.slice/crio-d050c588b4574451bf3d6857e2325c9a9b415ae464fd9de69bc898bf05d06d52 WatchSource:0}: Error finding container d050c588b4574451bf3d6857e2325c9a9b415ae464fd9de69bc898bf05d06d52: Status 404 returned error can't find the container with id d050c588b4574451bf3d6857e2325c9a9b415ae464fd9de69bc898bf05d06d52 Sep 11 15:31:30.331978 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:30.331939 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" event={"ID":"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc","Type":"ContainerStarted","Data":"c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226"} Sep 11 15:31:30.332394 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:30.332003 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" event={"ID":"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc","Type":"ContainerStarted","Data":"d050c588b4574451bf3d6857e2325c9a9b415ae464fd9de69bc898bf05d06d52"} Sep 11 15:31:30.332394 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:30.332033 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:31:30.349215 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:30.349169 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" podStartSLOduration=2.34915397 podStartE2EDuration="2.34915397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:31:30.346972831 +0000 UTC m=+1372.392297815" watchObservedRunningTime="2026-09-11 15:31:30.34915397 +0000 UTC m=+1372.394478950" Sep 11 15:31:38.707976 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:38.707941 2573 scope.go:117] "RemoveContainer" containerID="9e7520361dfa90bca1970752f0fe3032195cfb0ebb23a285c2c27407df2c0259" Sep 11 15:31:41.339663 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:31:41.339632 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:32:03.283335 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.283301 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t"] Sep 11 15:32:03.288326 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.288308 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:32:03.292169 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.292150 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-63c85a-dockercfg-rt4hz\"" Sep 11 15:32:03.292500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.292483 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-63c85a\"" Sep 11 15:32:03.299455 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.299421 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t"] Sep 11 15:32:03.307412 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.307391 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn"] Sep 11 15:32:03.311378 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.311356 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:32:03.322037 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.322009 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn"] Sep 11 15:32:03.408883 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.408858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-228wf\" (UniqueName: \"kubernetes.io/projected/f9b6871c-1172-4c9b-a31d-dede5d2a800d-kube-api-access-228wf\") pod \"payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t\" (UID: \"f9b6871c-1172-4c9b-a31d-dede5d2a800d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:32:03.408984 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.408937 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b6871c-1172-4c9b-a31d-dede5d2a800d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t\" (UID: \"f9b6871c-1172-4c9b-a31d-dede5d2a800d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:32:03.408984 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.408962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b508792-d9be-4c46-9841-5ebd74f3d882-plugins-config-volume\") pod \"payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn\" (UID: \"2b508792-d9be-4c46-9841-5ebd74f3d882\") " pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:32:03.409069 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.408988 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mh45\" (UniqueName: \"kubernetes.io/projected/2b508792-d9be-4c46-9841-5ebd74f3d882-kube-api-access-9mh45\") pod \"payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn\" (UID: \"2b508792-d9be-4c46-9841-5ebd74f3d882\") " pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:32:03.509747 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.509713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b6871c-1172-4c9b-a31d-dede5d2a800d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t\" (UID: \"f9b6871c-1172-4c9b-a31d-dede5d2a800d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:32:03.509896 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.509762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b508792-d9be-4c46-9841-5ebd74f3d882-plugins-config-volume\") pod \"payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn\" (UID: \"2b508792-d9be-4c46-9841-5ebd74f3d882\") " pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:32:03.509896 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.509804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mh45\" (UniqueName: \"kubernetes.io/projected/2b508792-d9be-4c46-9841-5ebd74f3d882-kube-api-access-9mh45\") pod \"payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn\" (UID: \"2b508792-d9be-4c46-9841-5ebd74f3d882\") " pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:32:03.510022 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.509904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-228wf\" (UniqueName: \"kubernetes.io/projected/f9b6871c-1172-4c9b-a31d-dede5d2a800d-kube-api-access-228wf\") pod \"payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t\" (UID: \"f9b6871c-1172-4c9b-a31d-dede5d2a800d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:32:03.510475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.510455 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b508792-d9be-4c46-9841-5ebd74f3d882-plugins-config-volume\") pod \"payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn\" (UID: \"2b508792-d9be-4c46-9841-5ebd74f3d882\") " pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:32:03.510541 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.510500 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b6871c-1172-4c9b-a31d-dede5d2a800d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t\" (UID: \"f9b6871c-1172-4c9b-a31d-dede5d2a800d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:32:03.517968 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.517950 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-228wf\" (UniqueName: \"kubernetes.io/projected/f9b6871c-1172-4c9b-a31d-dede5d2a800d-kube-api-access-228wf\") pod \"payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t\" (UID: \"f9b6871c-1172-4c9b-a31d-dede5d2a800d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:32:03.518483 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.518456 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mh45\" (UniqueName: \"kubernetes.io/projected/2b508792-d9be-4c46-9841-5ebd74f3d882-kube-api-access-9mh45\") pod \"payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn\" (UID: \"2b508792-d9be-4c46-9841-5ebd74f3d882\") " pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:32:03.600212 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.600191 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:32:03.624458 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.624437 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:32:03.743371 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.743333 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7340-379da2"] Sep 11 15:32:03.747162 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.747143 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-379da2" Sep 11 15:32:03.749547 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.749524 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6mbx9\"" Sep 11 15:32:03.751965 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.751944 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7340-379da2"] Sep 11 15:32:03.876173 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.876154 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t"] Sep 11 15:32:03.878385 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:32:03.878354 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b6871c_1172_4c9b_a31d_dede5d2a800d.slice/crio-82e4597b3abfc8874469d11bc3929af1a7dbd5c5a165822a472d6b993fdf1985 WatchSource:0}: Error finding container 82e4597b3abfc8874469d11bc3929af1a7dbd5c5a165822a472d6b993fdf1985: Status 404 returned error can't find the container with id 82e4597b3abfc8874469d11bc3929af1a7dbd5c5a165822a472d6b993fdf1985 Sep 11 15:32:03.907887 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.907866 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn"] Sep 11 15:32:03.911764 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:32:03.911742 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b508792_d9be_4c46_9841_5ebd74f3d882.slice/crio-ae5966e9df8c839ce35731eb135bbe31d8ef9a025824f001f68cbdcb8bfa467d WatchSource:0}: Error finding container ae5966e9df8c839ce35731eb135bbe31d8ef9a025824f001f68cbdcb8bfa467d: Status 404 returned error can't find the container with id ae5966e9df8c839ce35731eb135bbe31d8ef9a025824f001f68cbdcb8bfa467d Sep 11 15:32:03.913152 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:03.913135 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr4k6\" (UniqueName: \"kubernetes.io/projected/8f73e294-8388-40d1-83fe-4151d43b211f-kube-api-access-wr4k6\") pod \"test-curl-7340-379da2\" (UID: \"8f73e294-8388-40d1-83fe-4151d43b211f\") " pod="opendatahub/test-curl-7340-379da2" Sep 11 15:32:04.013940 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.013907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr4k6\" (UniqueName: \"kubernetes.io/projected/8f73e294-8388-40d1-83fe-4151d43b211f-kube-api-access-wr4k6\") pod \"test-curl-7340-379da2\" (UID: \"8f73e294-8388-40d1-83fe-4151d43b211f\") " pod="opendatahub/test-curl-7340-379da2" Sep 11 15:32:04.022647 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.022624 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr4k6\" (UniqueName: \"kubernetes.io/projected/8f73e294-8388-40d1-83fe-4151d43b211f-kube-api-access-wr4k6\") pod \"test-curl-7340-379da2\" (UID: \"8f73e294-8388-40d1-83fe-4151d43b211f\") " pod="opendatahub/test-curl-7340-379da2" Sep 11 15:32:04.058603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.058576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-379da2" Sep 11 15:32:04.333424 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.333399 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7340-379da2"] Sep 11 15:32:04.336440 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:32:04.336418 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f73e294_8388_40d1_83fe_4151d43b211f.slice/crio-00aa280b21d3f06293c450689c73d88886ab483ad9581b1f192dd4ffc31a89e4 WatchSource:0}: Error finding container 00aa280b21d3f06293c450689c73d88886ab483ad9581b1f192dd4ffc31a89e4: Status 404 returned error can't find the container with id 00aa280b21d3f06293c450689c73d88886ab483ad9581b1f192dd4ffc31a89e4 Sep 11 15:32:04.508732 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.508701 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7340-379da2" event={"ID":"8f73e294-8388-40d1-83fe-4151d43b211f","Type":"ContainerStarted","Data":"046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4"} Sep 11 15:32:04.508873 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.508742 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7340-379da2" event={"ID":"8f73e294-8388-40d1-83fe-4151d43b211f","Type":"ContainerStarted","Data":"00aa280b21d3f06293c450689c73d88886ab483ad9581b1f192dd4ffc31a89e4"} Sep 11 15:32:04.510342 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.510314 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" event={"ID":"2b508792-d9be-4c46-9841-5ebd74f3d882","Type":"ContainerStarted","Data":"872707f5e1f63718250750475bd8f99148d93f8fcae8edb3454d93fbf90771d8"} Sep 11 15:32:04.510471 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.510367 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" event={"ID":"2b508792-d9be-4c46-9841-5ebd74f3d882","Type":"ContainerStarted","Data":"ae5966e9df8c839ce35731eb135bbe31d8ef9a025824f001f68cbdcb8bfa467d"} Sep 11 15:32:04.510471 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.510399 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:32:04.511879 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.511860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" event={"ID":"f9b6871c-1172-4c9b-a31d-dede5d2a800d","Type":"ContainerStarted","Data":"12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8"} Sep 11 15:32:04.511986 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.511899 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" event={"ID":"f9b6871c-1172-4c9b-a31d-dede5d2a800d","Type":"ContainerStarted","Data":"82e4597b3abfc8874469d11bc3929af1a7dbd5c5a165822a472d6b993fdf1985"} Sep 11 15:32:04.511986 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.511935 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:32:04.524923 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.524878 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7340-379da2" podStartSLOduration=1.5248613130000002 podStartE2EDuration="1.524861313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:32:04.523241245 +0000 UTC m=+1406.568566225" watchObservedRunningTime="2026-09-11 15:32:04.524861313 +0000 UTC m=+1406.570186294" Sep 11 15:32:04.541183 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.540813 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" podStartSLOduration=1.5407945060000001 podStartE2EDuration="1.540794506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:32:04.538075732 +0000 UTC m=+1406.583400713" watchObservedRunningTime="2026-09-11 15:32:04.540794506 +0000 UTC m=+1406.586119485" Sep 11 15:32:04.555890 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:04.555848 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" podStartSLOduration=1.555836029 podStartE2EDuration="1.555836029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:32:04.553866589 +0000 UTC m=+1406.599191571" watchObservedRunningTime="2026-09-11 15:32:04.555836029 +0000 UTC m=+1406.601161005" Sep 11 15:32:05.108332 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.108304 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7340-379da2"] Sep 11 15:32:05.111613 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.111590 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7340-379da2"] Sep 11 15:32:05.229310 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.229279 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7340-da4aae"] Sep 11 15:32:05.229723 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.229710 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f73e294-8388-40d1-83fe-4151d43b211f" containerName="test-curl-7340-379da2" Sep 11 15:32:05.229803 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.229725 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f73e294-8388-40d1-83fe-4151d43b211f" containerName="test-curl-7340-379da2" Sep 11 15:32:05.229845 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.229826 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f73e294-8388-40d1-83fe-4151d43b211f" containerName="test-curl-7340-379da2" Sep 11 15:32:05.233075 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.233060 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-da4aae" Sep 11 15:32:05.238084 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.238060 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7340-da4aae"] Sep 11 15:32:05.325327 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.325297 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgqf7\" (UniqueName: \"kubernetes.io/projected/c8160754-90aa-4fce-b7c7-222e844e9324-kube-api-access-hgqf7\") pod \"test-curl-7340-da4aae\" (UID: \"c8160754-90aa-4fce-b7c7-222e844e9324\") " pod="opendatahub/test-curl-7340-da4aae" Sep 11 15:32:05.426635 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.426566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgqf7\" (UniqueName: \"kubernetes.io/projected/c8160754-90aa-4fce-b7c7-222e844e9324-kube-api-access-hgqf7\") pod \"test-curl-7340-da4aae\" (UID: \"c8160754-90aa-4fce-b7c7-222e844e9324\") " pod="opendatahub/test-curl-7340-da4aae" Sep 11 15:32:05.434387 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.434358 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgqf7\" (UniqueName: \"kubernetes.io/projected/c8160754-90aa-4fce-b7c7-222e844e9324-kube-api-access-hgqf7\") pod \"test-curl-7340-da4aae\" (UID: \"c8160754-90aa-4fce-b7c7-222e844e9324\") " pod="opendatahub/test-curl-7340-da4aae" Sep 11 15:32:05.516948 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:32:05.516919 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7340-379da2" container="test-curl-7340-379da2" Sep 11 15:32:05.544760 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.544740 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-da4aae" Sep 11 15:32:05.815730 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:05.815704 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7340-da4aae"] Sep 11 15:32:05.818140 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:32:05.818112 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8160754_90aa_4fce_b7c7_222e844e9324.slice/crio-a054c32939ce6ee0b938f755f4b5b725e4ce6b82b1f689d637affb5dfcc613df WatchSource:0}: Error finding container a054c32939ce6ee0b938f755f4b5b725e4ce6b82b1f689d637affb5dfcc613df: Status 404 returned error can't find the container with id a054c32939ce6ee0b938f755f4b5b725e4ce6b82b1f689d637affb5dfcc613df Sep 11 15:32:06.523160 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:06.523130 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7340-da4aae" event={"ID":"c8160754-90aa-4fce-b7c7-222e844e9324","Type":"ContainerStarted","Data":"f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e"} Sep 11 15:32:06.523535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:06.523173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7340-da4aae" event={"ID":"c8160754-90aa-4fce-b7c7-222e844e9324","Type":"ContainerStarted","Data":"a054c32939ce6ee0b938f755f4b5b725e4ce6b82b1f689d637affb5dfcc613df"} Sep 11 15:32:06.523535 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:06.523164 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7340-379da2" podUID="8f73e294-8388-40d1-83fe-4151d43b211f" containerName="test-curl-7340-379da2" containerID="cri-o://046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4" gracePeriod=2 Sep 11 15:32:06.538684 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:06.538639 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7340-da4aae" podStartSLOduration=1.538625667 podStartE2EDuration="1.538625667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:32:06.536700834 +0000 UTC m=+1408.582025816" watchObservedRunningTime="2026-09-11 15:32:06.538625667 +0000 UTC m=+1408.583950666" Sep 11 15:32:07.136404 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:07.136369 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7340-da4aae"] Sep 11 15:32:07.140365 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:07.140341 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7340-da4aae"] Sep 11 15:32:07.528394 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:32:07.528364 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7340-da4aae" container="test-curl-7340-da4aae" Sep 11 15:32:08.534078 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:08.534045 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7340-da4aae" podUID="c8160754-90aa-4fce-b7c7-222e844e9324" containerName="test-curl-7340-da4aae" containerID="cri-o://f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e" gracePeriod=2 Sep 11 15:32:08.692378 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:08.692348 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-379da2" Sep 11 15:32:08.857997 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:08.857975 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr4k6\" (UniqueName: \"kubernetes.io/projected/8f73e294-8388-40d1-83fe-4151d43b211f-kube-api-access-wr4k6\") pod \"8f73e294-8388-40d1-83fe-4151d43b211f\" (UID: \"8f73e294-8388-40d1-83fe-4151d43b211f\") " Sep 11 15:32:08.859987 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:08.859961 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f73e294-8388-40d1-83fe-4151d43b211f-kube-api-access-wr4k6" (OuterVolumeSpecName: "kube-api-access-wr4k6") pod "8f73e294-8388-40d1-83fe-4151d43b211f" (UID: "8f73e294-8388-40d1-83fe-4151d43b211f"). InnerVolumeSpecName "kube-api-access-wr4k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:32:08.959429 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:08.959402 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr4k6\" (UniqueName: \"kubernetes.io/projected/8f73e294-8388-40d1-83fe-4151d43b211f-kube-api-access-wr4k6\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:09.171592 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:09.171525 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2"] Sep 11 15:32:09.540667 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:09.540571 2573 generic.go:356] "Generic (PLEG): container finished" podID="8f73e294-8388-40d1-83fe-4151d43b211f" containerID="046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4" exitCode=137 Sep 11 15:32:09.540667 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:09.540663 2573 scope.go:117] "RemoveContainer" containerID="046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4" Sep 11 15:32:09.541195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:09.540852 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-379da2" Sep 11 15:32:09.551880 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:32:09.551735 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7340-379da2\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="8f73e294-8388-40d1-83fe-4151d43b211f" pod="opendatahub/test-curl-7340-379da2" Sep 11 15:32:09.551951 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:09.551890 2573 scope.go:117] "RemoveContainer" containerID="046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4" Sep 11 15:32:09.552180 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:32:09.552161 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4\": container with ID starting with 046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4 not found: ID does not exist" containerID="046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4" Sep 11 15:32:09.552253 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:09.552190 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4"} err="failed to get container status \"046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4\": rpc error: code = NotFound desc = could not find container \"046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4\": container with ID starting with 046acde027e5fe79e1aa6dda7b925694fec98721cd5b0b329d16a7794bd742f4 not found: ID does not exist" Sep 11 15:32:10.493902 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:10.493864 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f73e294-8388-40d1-83fe-4151d43b211f" path="/var/lib/kubelet/pods/8f73e294-8388-40d1-83fe-4151d43b211f/volumes" Sep 11 15:32:10.676477 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:10.676456 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-da4aae" Sep 11 15:32:10.776978 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:10.776921 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgqf7\" (UniqueName: \"kubernetes.io/projected/c8160754-90aa-4fce-b7c7-222e844e9324-kube-api-access-hgqf7\") pod \"c8160754-90aa-4fce-b7c7-222e844e9324\" (UID: \"c8160754-90aa-4fce-b7c7-222e844e9324\") " Sep 11 15:32:10.778960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:10.778935 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8160754-90aa-4fce-b7c7-222e844e9324-kube-api-access-hgqf7" (OuterVolumeSpecName: "kube-api-access-hgqf7") pod "c8160754-90aa-4fce-b7c7-222e844e9324" (UID: "c8160754-90aa-4fce-b7c7-222e844e9324"). InnerVolumeSpecName "kube-api-access-hgqf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:32:10.878105 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:10.878079 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgqf7\" (UniqueName: \"kubernetes.io/projected/c8160754-90aa-4fce-b7c7-222e844e9324-kube-api-access-hgqf7\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:11.553557 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:11.553524 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8160754-90aa-4fce-b7c7-222e844e9324" containerID="f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e" exitCode=137 Sep 11 15:32:11.553557 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:11.553554 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7340-da4aae" Sep 11 15:32:11.553811 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:11.553620 2573 scope.go:117] "RemoveContainer" containerID="f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e" Sep 11 15:32:11.562692 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:11.562674 2573 scope.go:117] "RemoveContainer" containerID="f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e" Sep 11 15:32:11.563027 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:32:11.563006 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e\": container with ID starting with f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e not found: ID does not exist" containerID="f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e" Sep 11 15:32:11.563108 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:11.563035 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e"} err="failed to get container status \"f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e\": rpc error: code = NotFound desc = could not find container \"f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e\": container with ID starting with f6e57d02a9ecc36aee2fd380d32bbe056a399c16b694be8d1d0034f82f15150e not found: ID does not exist" Sep 11 15:32:11.563212 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:32:11.563195 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7340-da4aae\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="c8160754-90aa-4fce-b7c7-222e844e9324" pod="opendatahub/test-curl-7340-da4aae" Sep 11 15:32:12.493913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:12.493878 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8160754-90aa-4fce-b7c7-222e844e9324" path="/var/lib/kubelet/pods/c8160754-90aa-4fce-b7c7-222e844e9324/volumes" Sep 11 15:32:15.519251 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:15.519217 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:32:15.519813 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:15.519418 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:32:17.500890 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.500855 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw"] Sep 11 15:32:17.501286 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.501274 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8160754-90aa-4fce-b7c7-222e844e9324" containerName="test-curl-7340-da4aae" Sep 11 15:32:17.501342 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.501289 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8160754-90aa-4fce-b7c7-222e844e9324" containerName="test-curl-7340-da4aae" Sep 11 15:32:17.501380 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.501371 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8160754-90aa-4fce-b7c7-222e844e9324" containerName="test-curl-7340-da4aae" Sep 11 15:32:17.505987 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.505970 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.510056 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.510032 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-397e40\"/\"rate-test-model-397e40-kserve-self-signed-certs\"" Sep 11 15:32:17.519112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.519093 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw"] Sep 11 15:32:17.630539 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.630504 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-home\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.630711 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.630569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tmp-dir\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.630711 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.630603 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tls-certs\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.630711 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.630696 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-dshm\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.630879 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.630742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-model-cache\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.630879 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.630836 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6gk\" (UniqueName: \"kubernetes.io/projected/57e6ebbf-766b-47a2-972a-0adae9f6ccda-kube-api-access-zn6gk\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.731489 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.731459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6gk\" (UniqueName: \"kubernetes.io/projected/57e6ebbf-766b-47a2-972a-0adae9f6ccda-kube-api-access-zn6gk\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.731639 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.731506 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-home\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.731639 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.731537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tmp-dir\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.731639 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.731555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tls-certs\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.731639 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.731575 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-dshm\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.731639 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.731601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-model-cache\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.731995 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.731973 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tmp-dir\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.732058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.732022 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-model-cache\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.732104 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.732068 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-home\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.733965 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.733944 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-dshm\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.734331 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.734312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tls-certs\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.739794 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.739749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6gk\" (UniqueName: \"kubernetes.io/projected/57e6ebbf-766b-47a2-972a-0adae9f6ccda-kube-api-access-zn6gk\") pod \"rate-test-model-397e40-kserve-5548cb49c5-qvwbw\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:17.815451 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:17.815397 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:18.088370 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:18.088345 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw"] Sep 11 15:32:18.593088 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:18.593049 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" event={"ID":"57e6ebbf-766b-47a2-972a-0adae9f6ccda","Type":"ContainerStarted","Data":"443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9"} Sep 11 15:32:18.593456 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:18.593093 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" event={"ID":"57e6ebbf-766b-47a2-972a-0adae9f6ccda","Type":"ContainerStarted","Data":"b366307cf8364ae288dd2babd9a6792c89853590b8e4edd8c0669d74599be556"} Sep 11 15:32:18.611682 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:18.611631 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" podStartSLOduration=1.611616752 podStartE2EDuration="1.611616752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:32:18.609827366 +0000 UTC m=+1420.655152340" watchObservedRunningTime="2026-09-11 15:32:18.611616752 +0000 UTC m=+1420.656941732" Sep 11 15:32:24.191081 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.191037 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" podUID="f0eb8228-53a8-4200-b725-2dad54755b57" containerName="main" containerID="cri-o://ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000" gracePeriod=45 Sep 11 15:32:24.442887 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.442833 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:32:24.484644 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.484616 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0eb8228-53a8-4200-b725-2dad54755b57-tls-certs\") pod \"f0eb8228-53a8-4200-b725-2dad54755b57\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " Sep 11 15:32:24.484753 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.484680 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-model-cache\") pod \"f0eb8228-53a8-4200-b725-2dad54755b57\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " Sep 11 15:32:24.484753 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.484702 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-tmp-dir\") pod \"f0eb8228-53a8-4200-b725-2dad54755b57\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " Sep 11 15:32:24.484862 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.484753 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2prhp\" (UniqueName: \"kubernetes.io/projected/f0eb8228-53a8-4200-b725-2dad54755b57-kube-api-access-2prhp\") pod \"f0eb8228-53a8-4200-b725-2dad54755b57\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " Sep 11 15:32:24.484862 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.484788 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-home\") pod \"f0eb8228-53a8-4200-b725-2dad54755b57\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " Sep 11 15:32:24.484862 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.484847 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-dshm\") pod \"f0eb8228-53a8-4200-b725-2dad54755b57\" (UID: \"f0eb8228-53a8-4200-b725-2dad54755b57\") " Sep 11 15:32:24.485004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.484977 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0eb8228-53a8-4200-b725-2dad54755b57" (UID: "f0eb8228-53a8-4200-b725-2dad54755b57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:32:24.485064 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.484992 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0eb8228-53a8-4200-b725-2dad54755b57" (UID: "f0eb8228-53a8-4200-b725-2dad54755b57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:32:24.485206 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.485180 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-home" (OuterVolumeSpecName: "home") pod "f0eb8228-53a8-4200-b725-2dad54755b57" (UID: "f0eb8228-53a8-4200-b725-2dad54755b57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:32:24.485276 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.485254 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:24.485340 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.485276 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-tmp-dir\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:24.487033 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.486999 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0eb8228-53a8-4200-b725-2dad54755b57-kube-api-access-2prhp" (OuterVolumeSpecName: "kube-api-access-2prhp") pod "f0eb8228-53a8-4200-b725-2dad54755b57" (UID: "f0eb8228-53a8-4200-b725-2dad54755b57"). InnerVolumeSpecName "kube-api-access-2prhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:32:24.487126 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.487032 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-dshm" (OuterVolumeSpecName: "dshm") pod "f0eb8228-53a8-4200-b725-2dad54755b57" (UID: "f0eb8228-53a8-4200-b725-2dad54755b57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:32:24.487126 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.487115 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0eb8228-53a8-4200-b725-2dad54755b57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0eb8228-53a8-4200-b725-2dad54755b57" (UID: "f0eb8228-53a8-4200-b725-2dad54755b57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:32:24.585953 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.585930 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2prhp\" (UniqueName: \"kubernetes.io/projected/f0eb8228-53a8-4200-b725-2dad54755b57-kube-api-access-2prhp\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:24.585953 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.585952 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-home\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:24.586120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.585962 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0eb8228-53a8-4200-b725-2dad54755b57-dshm\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:24.586120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.585970 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0eb8228-53a8-4200-b725-2dad54755b57-tls-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:24.633217 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.633191 2573 generic.go:356] "Generic (PLEG): container finished" podID="f0eb8228-53a8-4200-b725-2dad54755b57" containerID="ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000" exitCode=0 Sep 11 15:32:24.633339 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.633250 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" Sep 11 15:32:24.633339 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.633259 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" event={"ID":"f0eb8228-53a8-4200-b725-2dad54755b57","Type":"ContainerDied","Data":"ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000"} Sep 11 15:32:24.633339 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.633310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2" event={"ID":"f0eb8228-53a8-4200-b725-2dad54755b57","Type":"ContainerDied","Data":"db5e784518dcfaa8d86ba70a8fb0a068f76a38290174a7e82fe0d2b5db98570c"} Sep 11 15:32:24.633339 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.633334 2573 scope.go:117] "RemoveContainer" containerID="ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000" Sep 11 15:32:24.642069 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.642043 2573 scope.go:117] "RemoveContainer" containerID="ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000" Sep 11 15:32:24.642343 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:32:24.642323 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000\": container with ID starting with ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000 not found: ID does not exist" containerID="ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000" Sep 11 15:32:24.642392 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.642351 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000"} err="failed to get container status \"ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000\": rpc error: code = NotFound desc = could not find container \"ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000\": container with ID starting with ab6d75db918f7a63875b8b275a117f7485c5e812088f2e72fca8b7a6656f5000 not found: ID does not exist" Sep 11 15:32:24.650614 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.650594 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2"] Sep 11 15:32:24.656326 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:24.656307 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/sub-test-model-397e40-kserve-84df859fcf-kgrq2"] Sep 11 15:32:26.493100 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:26.493067 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0eb8228-53a8-4200-b725-2dad54755b57" path="/var/lib/kubelet/pods/f0eb8228-53a8-4200-b725-2dad54755b57/volumes" Sep 11 15:32:27.815892 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:27.815860 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:27.816261 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:27.815904 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:27.828423 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:27.828398 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:28.666291 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:28.666260 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:32:31.731342 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.731273 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957"] Sep 11 15:32:31.731751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.731735 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0eb8228-53a8-4200-b725-2dad54755b57" containerName="main" Sep 11 15:32:31.731822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.731753 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0eb8228-53a8-4200-b725-2dad54755b57" containerName="main" Sep 11 15:32:31.731863 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.731853 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0eb8228-53a8-4200-b725-2dad54755b57" containerName="main" Sep 11 15:32:31.736210 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.736195 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" Sep 11 15:32:31.738620 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.738599 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:32:31.738620 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.738607 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:32:31.742792 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.742751 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957"] Sep 11 15:32:31.840269 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.840241 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082b4b93-5d99-43f6-a767-b3012498f15d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-808212-8m957\" (UID: \"082b4b93-5d99-43f6-a767-b3012498f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" Sep 11 15:32:31.941405 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.941376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082b4b93-5d99-43f6-a767-b3012498f15d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-808212-8m957\" (UID: \"082b4b93-5d99-43f6-a767-b3012498f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" Sep 11 15:32:31.941987 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:31.941970 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082b4b93-5d99-43f6-a767-b3012498f15d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-808212-8m957\" (UID: \"082b4b93-5d99-43f6-a767-b3012498f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" Sep 11 15:32:32.048602 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:32.048544 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" Sep 11 15:32:32.319889 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:32.319858 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957"] Sep 11 15:32:32.321527 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:32:32.321503 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082b4b93_5d99_43f6_a767_b3012498f15d.slice/crio-7533c14ca3cd08bd7acc487100eb8f62766c9995d4244b77524135af069b4092 WatchSource:0}: Error finding container 7533c14ca3cd08bd7acc487100eb8f62766c9995d4244b77524135af069b4092: Status 404 returned error can't find the container with id 7533c14ca3cd08bd7acc487100eb8f62766c9995d4244b77524135af069b4092 Sep 11 15:32:32.679426 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:32.679395 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" event={"ID":"082b4b93-5d99-43f6-a767-b3012498f15d","Type":"ContainerStarted","Data":"de8c89d69e1c24f1faebc0b76e95b3b78d47d74133103f3998f29c90bf3abfcc"} Sep 11 15:32:32.679580 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:32.679445 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" event={"ID":"082b4b93-5d99-43f6-a767-b3012498f15d","Type":"ContainerStarted","Data":"7533c14ca3cd08bd7acc487100eb8f62766c9995d4244b77524135af069b4092"} Sep 11 15:32:32.710893 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:32.710846 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" podStartSLOduration=1.7108322390000001 podStartE2EDuration="1.710832239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:32:32.709023235 +0000 UTC m=+1434.754348239" watchObservedRunningTime="2026-09-11 15:32:32.710832239 +0000 UTC m=+1434.756157265" Sep 11 15:32:33.686442 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:33.686407 2573 generic.go:356] "Generic (PLEG): container finished" podID="082b4b93-5d99-43f6-a767-b3012498f15d" containerID="de8c89d69e1c24f1faebc0b76e95b3b78d47d74133103f3998f29c90bf3abfcc" exitCode=0 Sep 11 15:32:33.686835 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:33.686497 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" event={"ID":"082b4b93-5d99-43f6-a767-b3012498f15d","Type":"ContainerDied","Data":"de8c89d69e1c24f1faebc0b76e95b3b78d47d74133103f3998f29c90bf3abfcc"} Sep 11 15:32:34.827500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:34.827478 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" Sep 11 15:32:34.866979 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:34.866947 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082b4b93-5d99-43f6-a767-b3012498f15d-maas-api-service-ca\") pod \"082b4b93-5d99-43f6-a767-b3012498f15d\" (UID: \"082b4b93-5d99-43f6-a767-b3012498f15d\") " Sep 11 15:32:34.867265 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:34.867240 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082b4b93-5d99-43f6-a767-b3012498f15d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "082b4b93-5d99-43f6-a767-b3012498f15d" (UID: "082b4b93-5d99-43f6-a767-b3012498f15d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:32:34.968283 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:34.968216 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082b4b93-5d99-43f6-a767-b3012498f15d-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:35.700012 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:35.699983 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" Sep 11 15:32:35.700194 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:35.700026 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957" event={"ID":"082b4b93-5d99-43f6-a767-b3012498f15d","Type":"ContainerDied","Data":"7533c14ca3cd08bd7acc487100eb8f62766c9995d4244b77524135af069b4092"} Sep 11 15:32:35.700194 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:35.700076 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7533c14ca3cd08bd7acc487100eb8f62766c9995d4244b77524135af069b4092" Sep 11 15:32:38.736735 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:38.736708 2573 scope.go:117] "RemoveContainer" containerID="b91312ce4f722735411a3c58ed8a90088b8b32eb529ba197668904eac2c7d1a5" Sep 11 15:32:38.745078 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:38.745057 2573 scope.go:117] "RemoveContainer" containerID="d4c032081d021d7631f9e234c6576e0244c0e66b81d79af4f9dc6372675aaf95" Sep 11 15:32:38.752649 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:38.752631 2573 scope.go:117] "RemoveContainer" containerID="96a208c9bb9ce9bf1f288296324dd77196672d5d955e6cd5ac12d6ff7a0d22ef" Sep 11 15:32:47.307289 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.307258 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq"] Sep 11 15:32:47.307886 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.307488 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" podUID="df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc" containerName="payload-processing" containerID="cri-o://c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226" gracePeriod=30 Sep 11 15:32:47.471887 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.471850 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957"] Sep 11 15:32:47.474107 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.474075 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-808212-8m957"] Sep 11 15:32:47.579921 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.579893 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:32:47.675537 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.675508 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-plugins-config-volume\") pod \"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc\" (UID: \"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc\") " Sep 11 15:32:47.675708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.675634 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmtq2\" (UniqueName: \"kubernetes.io/projected/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-kube-api-access-fmtq2\") pod \"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc\" (UID: \"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc\") " Sep 11 15:32:47.675937 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.675911 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc" (UID: "df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:32:47.677844 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.677823 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-kube-api-access-fmtq2" (OuterVolumeSpecName: "kube-api-access-fmtq2") pod "df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc" (UID: "df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc"). InnerVolumeSpecName "kube-api-access-fmtq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:32:47.763149 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.763119 2573 generic.go:356] "Generic (PLEG): container finished" podID="df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc" containerID="c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226" exitCode=0 Sep 11 15:32:47.763253 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.763162 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" Sep 11 15:32:47.763253 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.763203 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" event={"ID":"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc","Type":"ContainerDied","Data":"c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226"} Sep 11 15:32:47.763340 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.763256 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq" event={"ID":"df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc","Type":"ContainerDied","Data":"d050c588b4574451bf3d6857e2325c9a9b415ae464fd9de69bc898bf05d06d52"} Sep 11 15:32:47.763340 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.763280 2573 scope.go:117] "RemoveContainer" containerID="c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226" Sep 11 15:32:47.772851 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.772832 2573 scope.go:117] "RemoveContainer" containerID="c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226" Sep 11 15:32:47.773124 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:32:47.773107 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226\": container with ID starting with c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226 not found: ID does not exist" containerID="c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226" Sep 11 15:32:47.773172 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.773132 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226"} err="failed to get container status \"c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226\": rpc error: code = NotFound desc = could not find container \"c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226\": container with ID starting with c296060d8df737481c804a893d01ed2a7ffb27135454d551e98c5f1481de4226 not found: ID does not exist" Sep 11 15:32:47.776593 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.776575 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmtq2\" (UniqueName: \"kubernetes.io/projected/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-kube-api-access-fmtq2\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:47.776653 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.776596 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:47.786969 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.786943 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq"] Sep 11 15:32:47.791314 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:47.791293 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-808212-79874d56df-gdhwq"] Sep 11 15:32:48.499751 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:48.499717 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="082b4b93-5d99-43f6-a767-b3012498f15d" path="/var/lib/kubelet/pods/082b4b93-5d99-43f6-a767-b3012498f15d/volumes" Sep 11 15:32:48.500175 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:48.500159 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc" path="/var/lib/kubelet/pods/df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc/volumes" Sep 11 15:32:54.238120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.238081 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7"] Sep 11 15:32:54.238704 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.238682 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc" containerName="payload-processing" Sep 11 15:32:54.238749 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.238711 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc" containerName="payload-processing" Sep 11 15:32:54.238749 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.238738 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082b4b93-5d99-43f6-a767-b3012498f15d" containerName="revoke-keys" Sep 11 15:32:54.238749 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.238748 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="082b4b93-5d99-43f6-a767-b3012498f15d" containerName="revoke-keys" Sep 11 15:32:54.238880 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.238873 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="082b4b93-5d99-43f6-a767-b3012498f15d" containerName="revoke-keys" Sep 11 15:32:54.238914 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.238889 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9be4f5-d6aa-4a94-b6ff-c20144d0f5bc" containerName="payload-processing" Sep 11 15:32:54.243637 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.243621 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7" Sep 11 15:32:54.246084 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.246058 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:32:54.246297 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.246065 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:32:54.248411 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.248385 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7"] Sep 11 15:32:54.331565 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.331534 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c885ae9-eee2-4d29-ac4a-2695718a1f38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7\" (UID: \"8c885ae9-eee2-4d29-ac4a-2695718a1f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7" Sep 11 15:32:54.432146 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.432120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c885ae9-eee2-4d29-ac4a-2695718a1f38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7\" (UID: \"8c885ae9-eee2-4d29-ac4a-2695718a1f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7" Sep 11 15:32:54.432817 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.432796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c885ae9-eee2-4d29-ac4a-2695718a1f38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7\" (UID: \"8c885ae9-eee2-4d29-ac4a-2695718a1f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7" Sep 11 15:32:54.556606 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.556550 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7" Sep 11 15:32:54.828607 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:54.828583 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7"] Sep 11 15:32:54.830762 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:32:54.830737 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c885ae9_eee2_4d29_ac4a_2695718a1f38.slice/crio-ac7730b17e380724ebaf187c6190b8a99a03e3fe66aece34b158a28b9f9f46b5 WatchSource:0}: Error finding container ac7730b17e380724ebaf187c6190b8a99a03e3fe66aece34b158a28b9f9f46b5: Status 404 returned error can't find the container with id ac7730b17e380724ebaf187c6190b8a99a03e3fe66aece34b158a28b9f9f46b5 Sep 11 15:32:55.805628 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:55.805552 2573 generic.go:356] "Generic (PLEG): container finished" podID="8c885ae9-eee2-4d29-ac4a-2695718a1f38" containerID="f10613510a546ffef4afa4e83568ca51c8b80409379012514e3419c4701892e7" exitCode=0 Sep 11 15:32:55.805999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:55.805635 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7" event={"ID":"8c885ae9-eee2-4d29-ac4a-2695718a1f38","Type":"ContainerDied","Data":"f10613510a546ffef4afa4e83568ca51c8b80409379012514e3419c4701892e7"} Sep 11 15:32:55.805999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:55.805690 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7" event={"ID":"8c885ae9-eee2-4d29-ac4a-2695718a1f38","Type":"ContainerStarted","Data":"ac7730b17e380724ebaf187c6190b8a99a03e3fe66aece34b158a28b9f9f46b5"} Sep 11 15:32:56.944874 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:56.944850 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7" Sep 11 15:32:57.054075 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:57.054049 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c885ae9-eee2-4d29-ac4a-2695718a1f38-maas-api-service-ca\") pod \"8c885ae9-eee2-4d29-ac4a-2695718a1f38\" (UID: \"8c885ae9-eee2-4d29-ac4a-2695718a1f38\") " Sep 11 15:32:57.054374 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:57.054354 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c885ae9-eee2-4d29-ac4a-2695718a1f38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c885ae9-eee2-4d29-ac4a-2695718a1f38" (UID: "8c885ae9-eee2-4d29-ac4a-2695718a1f38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:32:57.155025 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:57.155002 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c885ae9-eee2-4d29-ac4a-2695718a1f38-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:32:57.819265 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:57.819230 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7" Sep 11 15:32:57.819461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:57.819230 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7" event={"ID":"8c885ae9-eee2-4d29-ac4a-2695718a1f38","Type":"ContainerDied","Data":"ac7730b17e380724ebaf187c6190b8a99a03e3fe66aece34b158a28b9f9f46b5"} Sep 11 15:32:57.819461 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:32:57.819378 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac7730b17e380724ebaf187c6190b8a99a03e3fe66aece34b158a28b9f9f46b5" Sep 11 15:33:09.765481 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:09.765449 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn"] Sep 11 15:33:09.765988 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:09.765661 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" podUID="2b508792-d9be-4c46-9841-5ebd74f3d882" containerName="payload-processing" containerID="cri-o://872707f5e1f63718250750475bd8f99148d93f8fcae8edb3454d93fbf90771d8" gracePeriod=30 Sep 11 15:33:09.769086 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:09.769064 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t"] Sep 11 15:33:09.769373 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:09.769252 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" podUID="f9b6871c-1172-4c9b-a31d-dede5d2a800d" containerName="payload-pre-processing" containerID="cri-o://12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8" gracePeriod=30 Sep 11 15:33:09.887199 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:09.887156 2573 generic.go:356] "Generic (PLEG): container finished" podID="2b508792-d9be-4c46-9841-5ebd74f3d882" containerID="872707f5e1f63718250750475bd8f99148d93f8fcae8edb3454d93fbf90771d8" exitCode=0 Sep 11 15:33:09.887295 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:09.887192 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" event={"ID":"2b508792-d9be-4c46-9841-5ebd74f3d882","Type":"ContainerDied","Data":"872707f5e1f63718250750475bd8f99148d93f8fcae8edb3454d93fbf90771d8"} Sep 11 15:33:09.923391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:09.923344 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7"] Sep 11 15:33:09.925446 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:09.925418 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63c85a-zqtn7"] Sep 11 15:33:10.054744 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.054719 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:33:10.057914 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.057893 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:33:10.161679 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.161651 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b6871c-1172-4c9b-a31d-dede5d2a800d-plugins-config-volume\") pod \"f9b6871c-1172-4c9b-a31d-dede5d2a800d\" (UID: \"f9b6871c-1172-4c9b-a31d-dede5d2a800d\") " Sep 11 15:33:10.161841 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.161687 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mh45\" (UniqueName: \"kubernetes.io/projected/2b508792-d9be-4c46-9841-5ebd74f3d882-kube-api-access-9mh45\") pod \"2b508792-d9be-4c46-9841-5ebd74f3d882\" (UID: \"2b508792-d9be-4c46-9841-5ebd74f3d882\") " Sep 11 15:33:10.161841 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.161838 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-228wf\" (UniqueName: \"kubernetes.io/projected/f9b6871c-1172-4c9b-a31d-dede5d2a800d-kube-api-access-228wf\") pod \"f9b6871c-1172-4c9b-a31d-dede5d2a800d\" (UID: \"f9b6871c-1172-4c9b-a31d-dede5d2a800d\") " Sep 11 15:33:10.161937 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.161859 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b508792-d9be-4c46-9841-5ebd74f3d882-plugins-config-volume\") pod \"2b508792-d9be-4c46-9841-5ebd74f3d882\" (UID: \"2b508792-d9be-4c46-9841-5ebd74f3d882\") " Sep 11 15:33:10.162149 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.162098 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b6871c-1172-4c9b-a31d-dede5d2a800d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9b6871c-1172-4c9b-a31d-dede5d2a800d" (UID: "f9b6871c-1172-4c9b-a31d-dede5d2a800d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:33:10.162304 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.162278 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b508792-d9be-4c46-9841-5ebd74f3d882-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b508792-d9be-4c46-9841-5ebd74f3d882" (UID: "2b508792-d9be-4c46-9841-5ebd74f3d882"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:33:10.164087 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.164066 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b508792-d9be-4c46-9841-5ebd74f3d882-kube-api-access-9mh45" (OuterVolumeSpecName: "kube-api-access-9mh45") pod "2b508792-d9be-4c46-9841-5ebd74f3d882" (UID: "2b508792-d9be-4c46-9841-5ebd74f3d882"). InnerVolumeSpecName "kube-api-access-9mh45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:10.164184 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.164090 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b6871c-1172-4c9b-a31d-dede5d2a800d-kube-api-access-228wf" (OuterVolumeSpecName: "kube-api-access-228wf") pod "f9b6871c-1172-4c9b-a31d-dede5d2a800d" (UID: "f9b6871c-1172-4c9b-a31d-dede5d2a800d"). InnerVolumeSpecName "kube-api-access-228wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:10.262499 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.262477 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b6871c-1172-4c9b-a31d-dede5d2a800d-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:10.262499 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.262498 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mh45\" (UniqueName: \"kubernetes.io/projected/2b508792-d9be-4c46-9841-5ebd74f3d882-kube-api-access-9mh45\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:10.262637 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.262509 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-228wf\" (UniqueName: \"kubernetes.io/projected/f9b6871c-1172-4c9b-a31d-dede5d2a800d-kube-api-access-228wf\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:10.262637 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.262518 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b508792-d9be-4c46-9841-5ebd74f3d882-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:10.493973 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.493942 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c885ae9-eee2-4d29-ac4a-2695718a1f38" path="/var/lib/kubelet/pods/8c885ae9-eee2-4d29-ac4a-2695718a1f38/volumes" Sep 11 15:33:10.894058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.894030 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" Sep 11 15:33:10.894432 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.894034 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn" event={"ID":"2b508792-d9be-4c46-9841-5ebd74f3d882","Type":"ContainerDied","Data":"ae5966e9df8c839ce35731eb135bbe31d8ef9a025824f001f68cbdcb8bfa467d"} Sep 11 15:33:10.894432 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.894176 2573 scope.go:117] "RemoveContainer" containerID="872707f5e1f63718250750475bd8f99148d93f8fcae8edb3454d93fbf90771d8" Sep 11 15:33:10.895505 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.895480 2573 generic.go:356] "Generic (PLEG): container finished" podID="f9b6871c-1172-4c9b-a31d-dede5d2a800d" containerID="12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8" exitCode=0 Sep 11 15:33:10.895559 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.895514 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" Sep 11 15:33:10.895593 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.895552 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" event={"ID":"f9b6871c-1172-4c9b-a31d-dede5d2a800d","Type":"ContainerDied","Data":"12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8"} Sep 11 15:33:10.895630 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.895598 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t" event={"ID":"f9b6871c-1172-4c9b-a31d-dede5d2a800d","Type":"ContainerDied","Data":"82e4597b3abfc8874469d11bc3929af1a7dbd5c5a165822a472d6b993fdf1985"} Sep 11 15:33:10.903154 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.903139 2573 scope.go:117] "RemoveContainer" containerID="12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8" Sep 11 15:33:10.911063 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.911046 2573 scope.go:117] "RemoveContainer" containerID="12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8" Sep 11 15:33:10.911341 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:33:10.911317 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8\": container with ID starting with 12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8 not found: ID does not exist" containerID="12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8" Sep 11 15:33:10.911403 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.911348 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8"} err="failed to get container status \"12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8\": rpc error: code = NotFound desc = could not find container \"12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8\": container with ID starting with 12a4668a9a39d1c2645a5be3d8ad2c3af5ea8f57b9efc0779a86de0a04ac6fb8 not found: ID does not exist" Sep 11 15:33:10.914668 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.914648 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t"] Sep 11 15:33:10.920968 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.920938 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-63c85a-655564df86-sp96t"] Sep 11 15:33:10.930496 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.930476 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn"] Sep 11 15:33:10.934811 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:10.934767 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-63c85a-547d4b768f-89cpn"] Sep 11 15:33:12.492954 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:12.492924 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b508792-d9be-4c46-9841-5ebd74f3d882" path="/var/lib/kubelet/pods/2b508792-d9be-4c46-9841-5ebd74f3d882/volumes" Sep 11 15:33:12.493297 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:12.493264 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b6871c-1172-4c9b-a31d-dede5d2a800d" path="/var/lib/kubelet/pods/f9b6871c-1172-4c9b-a31d-dede5d2a800d/volumes" Sep 11 15:33:16.527211 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.527180 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg"] Sep 11 15:33:16.527806 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.527765 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c885ae9-eee2-4d29-ac4a-2695718a1f38" containerName="revoke-keys" Sep 11 15:33:16.527913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.527809 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c885ae9-eee2-4d29-ac4a-2695718a1f38" containerName="revoke-keys" Sep 11 15:33:16.527913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.527829 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b6871c-1172-4c9b-a31d-dede5d2a800d" containerName="payload-pre-processing" Sep 11 15:33:16.527913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.527838 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b6871c-1172-4c9b-a31d-dede5d2a800d" containerName="payload-pre-processing" Sep 11 15:33:16.527913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.527870 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b508792-d9be-4c46-9841-5ebd74f3d882" containerName="payload-processing" Sep 11 15:33:16.527913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.527879 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b508792-d9be-4c46-9841-5ebd74f3d882" containerName="payload-processing" Sep 11 15:33:16.528165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.527975 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b6871c-1172-4c9b-a31d-dede5d2a800d" containerName="payload-pre-processing" Sep 11 15:33:16.528165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.527986 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b508792-d9be-4c46-9841-5ebd74f3d882" containerName="payload-processing" Sep 11 15:33:16.528165 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.527995 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c885ae9-eee2-4d29-ac4a-2695718a1f38" containerName="revoke-keys" Sep 11 15:33:16.532979 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.532955 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg" Sep 11 15:33:16.535533 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.535511 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:33:16.535848 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.535824 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:33:16.537793 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.537751 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg"] Sep 11 15:33:16.614419 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.614398 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc37b06f-7b6e-4785-ba68-e29eceea1b9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg\" (UID: \"bc37b06f-7b6e-4785-ba68-e29eceea1b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg" Sep 11 15:33:16.715181 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.715156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc37b06f-7b6e-4785-ba68-e29eceea1b9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg\" (UID: \"bc37b06f-7b6e-4785-ba68-e29eceea1b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg" Sep 11 15:33:16.715700 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.715681 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc37b06f-7b6e-4785-ba68-e29eceea1b9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg\" (UID: \"bc37b06f-7b6e-4785-ba68-e29eceea1b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg" Sep 11 15:33:16.845867 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:16.845836 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg" Sep 11 15:33:17.117757 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:17.117732 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg"] Sep 11 15:33:17.119397 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:33:17.119363 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc37b06f_7b6e_4785_ba68_e29eceea1b9c.slice/crio-09f184978de65d7b04b5b67c7e335f21c84dc9a4a3d7eefde60c460d36da21ee WatchSource:0}: Error finding container 09f184978de65d7b04b5b67c7e335f21c84dc9a4a3d7eefde60c460d36da21ee: Status 404 returned error can't find the container with id 09f184978de65d7b04b5b67c7e335f21c84dc9a4a3d7eefde60c460d36da21ee Sep 11 15:33:17.944898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:17.944865 2573 generic.go:356] "Generic (PLEG): container finished" podID="bc37b06f-7b6e-4785-ba68-e29eceea1b9c" containerID="bfb7b09189abb48c43c98c7309335aa45afbce720e988b6c521d4e298340d616" exitCode=0 Sep 11 15:33:17.945204 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:17.944946 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg" event={"ID":"bc37b06f-7b6e-4785-ba68-e29eceea1b9c","Type":"ContainerDied","Data":"bfb7b09189abb48c43c98c7309335aa45afbce720e988b6c521d4e298340d616"} Sep 11 15:33:17.945204 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:17.944999 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg" event={"ID":"bc37b06f-7b6e-4785-ba68-e29eceea1b9c","Type":"ContainerStarted","Data":"09f184978de65d7b04b5b67c7e335f21c84dc9a4a3d7eefde60c460d36da21ee"} Sep 11 15:33:19.080696 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:19.080673 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg" Sep 11 15:33:19.236394 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:19.236323 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc37b06f-7b6e-4785-ba68-e29eceea1b9c-maas-api-service-ca\") pod \"bc37b06f-7b6e-4785-ba68-e29eceea1b9c\" (UID: \"bc37b06f-7b6e-4785-ba68-e29eceea1b9c\") " Sep 11 15:33:19.236710 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:19.236688 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc37b06f-7b6e-4785-ba68-e29eceea1b9c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc37b06f-7b6e-4785-ba68-e29eceea1b9c" (UID: "bc37b06f-7b6e-4785-ba68-e29eceea1b9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:33:19.337712 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:19.337685 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc37b06f-7b6e-4785-ba68-e29eceea1b9c-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:19.957149 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:19.957114 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg" event={"ID":"bc37b06f-7b6e-4785-ba68-e29eceea1b9c","Type":"ContainerDied","Data":"09f184978de65d7b04b5b67c7e335f21c84dc9a4a3d7eefde60c460d36da21ee"} Sep 11 15:33:19.957309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:19.957165 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09f184978de65d7b04b5b67c7e335f21c84dc9a4a3d7eefde60c460d36da21ee" Sep 11 15:33:19.957309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:19.957131 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg" Sep 11 15:33:26.539760 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:26.539728 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw"] Sep 11 15:33:27.245708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.245680 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28"] Sep 11 15:33:27.250454 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.250425 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a348b-29819010-xtc28"] Sep 11 15:33:27.272218 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.272180 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x"] Sep 11 15:33:27.272438 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.272407 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" podUID="87ab3b9b-a467-492b-ab28-6a4ce6c63752" containerName="payload-processing" containerID="cri-o://8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620" gracePeriod=30 Sep 11 15:33:27.280260 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.279676 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg"] Sep 11 15:33:27.280260 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.279908 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" podUID="6f7d0270-23ab-42dd-a63d-881a5b78cf56" containerName="payload-pre-processing" containerID="cri-o://3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f" gracePeriod=30 Sep 11 15:33:27.444650 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.443827 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg"] Sep 11 15:33:27.450085 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.450055 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a348b-zwqgg"] Sep 11 15:33:27.531501 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.531475 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:33:27.554129 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.554097 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:33:27.609744 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.609715 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ab3b9b-a467-492b-ab28-6a4ce6c63752-plugins-config-volume\") pod \"87ab3b9b-a467-492b-ab28-6a4ce6c63752\" (UID: \"87ab3b9b-a467-492b-ab28-6a4ce6c63752\") " Sep 11 15:33:27.609913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.609760 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhm7c\" (UniqueName: \"kubernetes.io/projected/87ab3b9b-a467-492b-ab28-6a4ce6c63752-kube-api-access-hhm7c\") pod \"87ab3b9b-a467-492b-ab28-6a4ce6c63752\" (UID: \"87ab3b9b-a467-492b-ab28-6a4ce6c63752\") " Sep 11 15:33:27.609913 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.609878 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj2lg\" (UniqueName: \"kubernetes.io/projected/6f7d0270-23ab-42dd-a63d-881a5b78cf56-kube-api-access-zj2lg\") pod \"6f7d0270-23ab-42dd-a63d-881a5b78cf56\" (UID: \"6f7d0270-23ab-42dd-a63d-881a5b78cf56\") " Sep 11 15:33:27.610040 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.609961 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7d0270-23ab-42dd-a63d-881a5b78cf56-plugins-config-volume\") pod \"6f7d0270-23ab-42dd-a63d-881a5b78cf56\" (UID: \"6f7d0270-23ab-42dd-a63d-881a5b78cf56\") " Sep 11 15:33:27.610171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.610148 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ab3b9b-a467-492b-ab28-6a4ce6c63752-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87ab3b9b-a467-492b-ab28-6a4ce6c63752" (UID: "87ab3b9b-a467-492b-ab28-6a4ce6c63752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:33:27.610307 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.610287 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ab3b9b-a467-492b-ab28-6a4ce6c63752-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:27.610380 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.610341 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7d0270-23ab-42dd-a63d-881a5b78cf56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f7d0270-23ab-42dd-a63d-881a5b78cf56" (UID: "6f7d0270-23ab-42dd-a63d-881a5b78cf56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:33:27.612224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.612200 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7d0270-23ab-42dd-a63d-881a5b78cf56-kube-api-access-zj2lg" (OuterVolumeSpecName: "kube-api-access-zj2lg") pod "6f7d0270-23ab-42dd-a63d-881a5b78cf56" (UID: "6f7d0270-23ab-42dd-a63d-881a5b78cf56"). InnerVolumeSpecName "kube-api-access-zj2lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:27.612309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.612232 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ab3b9b-a467-492b-ab28-6a4ce6c63752-kube-api-access-hhm7c" (OuterVolumeSpecName: "kube-api-access-hhm7c") pod "87ab3b9b-a467-492b-ab28-6a4ce6c63752" (UID: "87ab3b9b-a467-492b-ab28-6a4ce6c63752"). InnerVolumeSpecName "kube-api-access-hhm7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:27.711469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.711444 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7d0270-23ab-42dd-a63d-881a5b78cf56-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:27.711469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.711468 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhm7c\" (UniqueName: \"kubernetes.io/projected/87ab3b9b-a467-492b-ab28-6a4ce6c63752-kube-api-access-hhm7c\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:27.711635 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:27.711478 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj2lg\" (UniqueName: \"kubernetes.io/projected/6f7d0270-23ab-42dd-a63d-881a5b78cf56-kube-api-access-zj2lg\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:28.000875 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.000836 2573 generic.go:356] "Generic (PLEG): container finished" podID="6f7d0270-23ab-42dd-a63d-881a5b78cf56" containerID="3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f" exitCode=0 Sep 11 15:33:28.001058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.000883 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" Sep 11 15:33:28.001058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.000922 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" event={"ID":"6f7d0270-23ab-42dd-a63d-881a5b78cf56","Type":"ContainerDied","Data":"3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f"} Sep 11 15:33:28.001058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.001002 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg" event={"ID":"6f7d0270-23ab-42dd-a63d-881a5b78cf56","Type":"ContainerDied","Data":"eb90927c903ee1302c7629259712cff001646f1fbb5be967d4dc4beb7ac6ea61"} Sep 11 15:33:28.001058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.001035 2573 scope.go:117] "RemoveContainer" containerID="3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f" Sep 11 15:33:28.002312 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.002285 2573 generic.go:356] "Generic (PLEG): container finished" podID="87ab3b9b-a467-492b-ab28-6a4ce6c63752" containerID="8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620" exitCode=0 Sep 11 15:33:28.002430 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.002334 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" Sep 11 15:33:28.002430 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.002382 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" event={"ID":"87ab3b9b-a467-492b-ab28-6a4ce6c63752","Type":"ContainerDied","Data":"8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620"} Sep 11 15:33:28.002430 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.002424 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x" event={"ID":"87ab3b9b-a467-492b-ab28-6a4ce6c63752","Type":"ContainerDied","Data":"66b7ca192aaa009790c1c37402af96e52368f2cc458d86ad18efc972c999b348"} Sep 11 15:33:28.011135 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.011116 2573 scope.go:117] "RemoveContainer" containerID="3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f" Sep 11 15:33:28.011437 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:33:28.011414 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f\": container with ID starting with 3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f not found: ID does not exist" containerID="3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f" Sep 11 15:33:28.011508 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.011444 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f"} err="failed to get container status \"3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f\": rpc error: code = NotFound desc = could not find container \"3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f\": container with ID starting with 3610e724342b8056a018e2de9b1d71b5e7f170a9ed5e3ae67682b148735c5e7f not found: ID does not exist" Sep 11 15:33:28.011508 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.011459 2573 scope.go:117] "RemoveContainer" containerID="8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620" Sep 11 15:33:28.019765 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.019748 2573 scope.go:117] "RemoveContainer" containerID="8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620" Sep 11 15:33:28.020038 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:33:28.020014 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620\": container with ID starting with 8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620 not found: ID does not exist" containerID="8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620" Sep 11 15:33:28.020098 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.020046 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620"} err="failed to get container status \"8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620\": rpc error: code = NotFound desc = could not find container \"8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620\": container with ID starting with 8c5f0c335ac9dde4230062be39a649ced31d68448159a1d7948ecfae89f0b620 not found: ID does not exist" Sep 11 15:33:28.024834 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.024814 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg"] Sep 11 15:33:28.030387 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.030365 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a348b-7748d766c4-l96wg"] Sep 11 15:33:28.040453 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.040435 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x"] Sep 11 15:33:28.044402 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.044384 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a348b-6b5cf6f7b5-nm22x"] Sep 11 15:33:28.492910 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.492881 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f7d0270-23ab-42dd-a63d-881a5b78cf56" path="/var/lib/kubelet/pods/6f7d0270-23ab-42dd-a63d-881a5b78cf56/volumes" Sep 11 15:33:28.493241 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.493229 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ab3b9b-a467-492b-ab28-6a4ce6c63752" path="/var/lib/kubelet/pods/87ab3b9b-a467-492b-ab28-6a4ce6c63752/volumes" Sep 11 15:33:28.493509 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.493497 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc37b06f-7b6e-4785-ba68-e29eceea1b9c" path="/var/lib/kubelet/pods/bc37b06f-7b6e-4785-ba68-e29eceea1b9c/volumes" Sep 11 15:33:28.493742 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:28.493729 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c274876b-e3df-4fe3-adcd-3654dd677bb4" path="/var/lib/kubelet/pods/c274876b-e3df-4fe3-adcd-3654dd677bb4/volumes" Sep 11 15:33:34.530597 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.530566 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28"] Sep 11 15:33:34.531000 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.530956 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7d0270-23ab-42dd-a63d-881a5b78cf56" containerName="payload-pre-processing" Sep 11 15:33:34.531000 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.530969 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7d0270-23ab-42dd-a63d-881a5b78cf56" containerName="payload-pre-processing" Sep 11 15:33:34.531000 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.530980 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc37b06f-7b6e-4785-ba68-e29eceea1b9c" containerName="revoke-keys" Sep 11 15:33:34.531000 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.530985 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc37b06f-7b6e-4785-ba68-e29eceea1b9c" containerName="revoke-keys" Sep 11 15:33:34.531000 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.530995 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ab3b9b-a467-492b-ab28-6a4ce6c63752" containerName="payload-processing" Sep 11 15:33:34.531000 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.531001 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ab3b9b-a467-492b-ab28-6a4ce6c63752" containerName="payload-processing" Sep 11 15:33:34.531192 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.531067 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7d0270-23ab-42dd-a63d-881a5b78cf56" containerName="payload-pre-processing" Sep 11 15:33:34.531192 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.531077 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ab3b9b-a467-492b-ab28-6a4ce6c63752" containerName="payload-processing" Sep 11 15:33:34.531192 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.531083 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc37b06f-7b6e-4785-ba68-e29eceea1b9c" containerName="revoke-keys" Sep 11 15:33:34.537752 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.537729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28" Sep 11 15:33:34.540273 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.540076 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:33:34.540385 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.540342 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:33:34.544639 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.544614 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28"] Sep 11 15:33:34.567169 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.567147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67400989-7226-4ee7-bf8b-a771fd5a852f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28\" (UID: \"67400989-7226-4ee7-bf8b-a771fd5a852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28" Sep 11 15:33:34.668178 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.668151 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67400989-7226-4ee7-bf8b-a771fd5a852f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28\" (UID: \"67400989-7226-4ee7-bf8b-a771fd5a852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28" Sep 11 15:33:34.668743 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.668719 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67400989-7226-4ee7-bf8b-a771fd5a852f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28\" (UID: \"67400989-7226-4ee7-bf8b-a771fd5a852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28" Sep 11 15:33:34.852241 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:34.852213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28" Sep 11 15:33:35.124231 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:35.124206 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28"] Sep 11 15:33:35.126344 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:33:35.126319 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67400989_7226_4ee7_bf8b_a771fd5a852f.slice/crio-497a7a37209b6d7695a75c086d0e31251ae6c5c18ca4961d2c4310721c65e50b WatchSource:0}: Error finding container 497a7a37209b6d7695a75c086d0e31251ae6c5c18ca4961d2c4310721c65e50b: Status 404 returned error can't find the container with id 497a7a37209b6d7695a75c086d0e31251ae6c5c18ca4961d2c4310721c65e50b Sep 11 15:33:36.046528 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.046496 2573 generic.go:356] "Generic (PLEG): container finished" podID="67400989-7226-4ee7-bf8b-a771fd5a852f" containerID="648df85361abdc85fb2bc6e1bc6edb9ed30f6b91ef2adb3c15d39fdc3e69ecc7" exitCode=0 Sep 11 15:33:36.046904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.046546 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28" event={"ID":"67400989-7226-4ee7-bf8b-a771fd5a852f","Type":"ContainerDied","Data":"648df85361abdc85fb2bc6e1bc6edb9ed30f6b91ef2adb3c15d39fdc3e69ecc7"} Sep 11 15:33:36.046904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.046583 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28" event={"ID":"67400989-7226-4ee7-bf8b-a771fd5a852f","Type":"ContainerStarted","Data":"497a7a37209b6d7695a75c086d0e31251ae6c5c18ca4961d2c4310721c65e50b"} Sep 11 15:33:36.069021 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.068994 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n"] Sep 11 15:33:36.072720 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.072702 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:33:36.075029 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.075000 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8559ee-dockercfg-5k826\"" Sep 11 15:33:36.075137 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.075118 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8559ee\"" Sep 11 15:33:36.083250 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.083229 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n"] Sep 11 15:33:36.092939 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.092918 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr"] Sep 11 15:33:36.096735 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.096717 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:33:36.108853 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.108828 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr"] Sep 11 15:33:36.182135 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.182072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drl4m\" (UniqueName: \"kubernetes.io/projected/8759f770-68a2-4bd2-a561-bf4992564118-kube-api-access-drl4m\") pod \"payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n\" (UID: \"8759f770-68a2-4bd2-a561-bf4992564118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:33:36.182135 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.182113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d277cb-ce76-4d1b-bc15-a1f64a898f14-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr\" (UID: \"22d277cb-ce76-4d1b-bc15-a1f64a898f14\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:33:36.182309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.182179 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8759f770-68a2-4bd2-a561-bf4992564118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n\" (UID: \"8759f770-68a2-4bd2-a561-bf4992564118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:33:36.182309 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.182223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szw4m\" (UniqueName: \"kubernetes.io/projected/22d277cb-ce76-4d1b-bc15-a1f64a898f14-kube-api-access-szw4m\") pod \"payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr\" (UID: \"22d277cb-ce76-4d1b-bc15-a1f64a898f14\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:33:36.282946 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.282918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d277cb-ce76-4d1b-bc15-a1f64a898f14-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr\" (UID: \"22d277cb-ce76-4d1b-bc15-a1f64a898f14\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:33:36.283066 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.282967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8759f770-68a2-4bd2-a561-bf4992564118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n\" (UID: \"8759f770-68a2-4bd2-a561-bf4992564118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:33:36.283132 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.283061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szw4m\" (UniqueName: \"kubernetes.io/projected/22d277cb-ce76-4d1b-bc15-a1f64a898f14-kube-api-access-szw4m\") pod \"payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr\" (UID: \"22d277cb-ce76-4d1b-bc15-a1f64a898f14\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:33:36.283187 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.283160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drl4m\" (UniqueName: \"kubernetes.io/projected/8759f770-68a2-4bd2-a561-bf4992564118-kube-api-access-drl4m\") pod \"payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n\" (UID: \"8759f770-68a2-4bd2-a561-bf4992564118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:33:36.283650 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.283630 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8759f770-68a2-4bd2-a561-bf4992564118-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n\" (UID: \"8759f770-68a2-4bd2-a561-bf4992564118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:33:36.283738 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.283689 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d277cb-ce76-4d1b-bc15-a1f64a898f14-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr\" (UID: \"22d277cb-ce76-4d1b-bc15-a1f64a898f14\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:33:36.290694 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.290670 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szw4m\" (UniqueName: \"kubernetes.io/projected/22d277cb-ce76-4d1b-bc15-a1f64a898f14-kube-api-access-szw4m\") pod \"payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr\" (UID: \"22d277cb-ce76-4d1b-bc15-a1f64a898f14\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:33:36.291013 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.290993 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drl4m\" (UniqueName: \"kubernetes.io/projected/8759f770-68a2-4bd2-a561-bf4992564118-kube-api-access-drl4m\") pod \"payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n\" (UID: \"8759f770-68a2-4bd2-a561-bf4992564118\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:33:36.383188 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.383165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:33:36.410341 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.410318 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:33:36.654861 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.654837 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n"] Sep 11 15:33:36.656717 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:33:36.656688 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8759f770_68a2_4bd2_a561_bf4992564118.slice/crio-af4bc98bafe46ed31363d2554fdaff0eb01c1c09c7a5d8966d8a29948bda03fb WatchSource:0}: Error finding container af4bc98bafe46ed31363d2554fdaff0eb01c1c09c7a5d8966d8a29948bda03fb: Status 404 returned error can't find the container with id af4bc98bafe46ed31363d2554fdaff0eb01c1c09c7a5d8966d8a29948bda03fb Sep 11 15:33:36.688557 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:36.688532 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr"] Sep 11 15:33:36.690292 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:33:36.690256 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d277cb_ce76_4d1b_bc15_a1f64a898f14.slice/crio-2bd41d5f89aea4da0e7b22e0a3fc833f1a400d9f9b5a6517107e6d28fba76433 WatchSource:0}: Error finding container 2bd41d5f89aea4da0e7b22e0a3fc833f1a400d9f9b5a6517107e6d28fba76433: Status 404 returned error can't find the container with id 2bd41d5f89aea4da0e7b22e0a3fc833f1a400d9f9b5a6517107e6d28fba76433 Sep 11 15:33:37.054354 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.054273 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" event={"ID":"8759f770-68a2-4bd2-a561-bf4992564118","Type":"ContainerStarted","Data":"04544db77cd9ca71e1c0b607d89c0524a334be79884b57ac55f8825dbc02d410"} Sep 11 15:33:37.054739 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.054684 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:33:37.054824 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.054738 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" event={"ID":"8759f770-68a2-4bd2-a561-bf4992564118","Type":"ContainerStarted","Data":"af4bc98bafe46ed31363d2554fdaff0eb01c1c09c7a5d8966d8a29948bda03fb"} Sep 11 15:33:37.056251 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.056218 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" event={"ID":"22d277cb-ce76-4d1b-bc15-a1f64a898f14","Type":"ContainerStarted","Data":"8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880"} Sep 11 15:33:37.056366 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.056269 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" event={"ID":"22d277cb-ce76-4d1b-bc15-a1f64a898f14","Type":"ContainerStarted","Data":"2bd41d5f89aea4da0e7b22e0a3fc833f1a400d9f9b5a6517107e6d28fba76433"} Sep 11 15:33:37.071321 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.071267 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" podStartSLOduration=1.071247893 podStartE2EDuration="1.071247893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:33:37.069620675 +0000 UTC m=+1499.114945655" watchObservedRunningTime="2026-09-11 15:33:37.071247893 +0000 UTC m=+1499.116572876" Sep 11 15:33:37.088920 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.088864 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" podStartSLOduration=1.088846107 podStartE2EDuration="1.088846107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:33:37.085630513 +0000 UTC m=+1499.130955499" watchObservedRunningTime="2026-09-11 15:33:37.088846107 +0000 UTC m=+1499.134171088" Sep 11 15:33:37.203505 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.203477 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28" Sep 11 15:33:37.291713 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.291685 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67400989-7226-4ee7-bf8b-a771fd5a852f-maas-api-service-ca\") pod \"67400989-7226-4ee7-bf8b-a771fd5a852f\" (UID: \"67400989-7226-4ee7-bf8b-a771fd5a852f\") " Sep 11 15:33:37.292007 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.291984 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67400989-7226-4ee7-bf8b-a771fd5a852f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67400989-7226-4ee7-bf8b-a771fd5a852f" (UID: "67400989-7226-4ee7-bf8b-a771fd5a852f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:33:37.392697 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:37.392671 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67400989-7226-4ee7-bf8b-a771fd5a852f-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:38.062823 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:38.062797 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28" Sep 11 15:33:38.063191 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:38.062825 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28" event={"ID":"67400989-7226-4ee7-bf8b-a771fd5a852f","Type":"ContainerDied","Data":"497a7a37209b6d7695a75c086d0e31251ae6c5c18ca4961d2c4310721c65e50b"} Sep 11 15:33:38.063191 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:38.062882 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="497a7a37209b6d7695a75c086d0e31251ae6c5c18ca4961d2c4310721c65e50b" Sep 11 15:33:38.063191 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:38.063118 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:33:38.510316 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:38.510286 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:33:38.513161 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:38.513138 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:33:41.560227 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.560189 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" podUID="57e6ebbf-766b-47a2-972a-0adae9f6ccda" containerName="main" containerID="cri-o://443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9" gracePeriod=45 Sep 11 15:33:41.802695 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.802673 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:33:41.935675 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.935646 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tls-certs\") pod \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " Sep 11 15:33:41.935857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.935692 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-home\") pod \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " Sep 11 15:33:41.935857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.935741 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-model-cache\") pod \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " Sep 11 15:33:41.935857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.935812 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn6gk\" (UniqueName: \"kubernetes.io/projected/57e6ebbf-766b-47a2-972a-0adae9f6ccda-kube-api-access-zn6gk\") pod \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " Sep 11 15:33:41.936026 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.935860 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-dshm\") pod \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " Sep 11 15:33:41.936026 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.935905 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tmp-dir\") pod \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\" (UID: \"57e6ebbf-766b-47a2-972a-0adae9f6ccda\") " Sep 11 15:33:41.936135 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.936013 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-home" (OuterVolumeSpecName: "home") pod "57e6ebbf-766b-47a2-972a-0adae9f6ccda" (UID: "57e6ebbf-766b-47a2-972a-0adae9f6ccda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:41.936135 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.936038 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-model-cache" (OuterVolumeSpecName: "model-cache") pod "57e6ebbf-766b-47a2-972a-0adae9f6ccda" (UID: "57e6ebbf-766b-47a2-972a-0adae9f6ccda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:41.936245 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.936185 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-home\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:41.936245 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.936207 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:41.936245 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.936223 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57e6ebbf-766b-47a2-972a-0adae9f6ccda" (UID: "57e6ebbf-766b-47a2-972a-0adae9f6ccda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:41.938171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.938146 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57e6ebbf-766b-47a2-972a-0adae9f6ccda" (UID: "57e6ebbf-766b-47a2-972a-0adae9f6ccda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:33:41.938171 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.938159 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e6ebbf-766b-47a2-972a-0adae9f6ccda-kube-api-access-zn6gk" (OuterVolumeSpecName: "kube-api-access-zn6gk") pod "57e6ebbf-766b-47a2-972a-0adae9f6ccda" (UID: "57e6ebbf-766b-47a2-972a-0adae9f6ccda"). InnerVolumeSpecName "kube-api-access-zn6gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:41.938341 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:41.938148 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-dshm" (OuterVolumeSpecName: "dshm") pod "57e6ebbf-766b-47a2-972a-0adae9f6ccda" (UID: "57e6ebbf-766b-47a2-972a-0adae9f6ccda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:42.036641 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.036611 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn6gk\" (UniqueName: \"kubernetes.io/projected/57e6ebbf-766b-47a2-972a-0adae9f6ccda-kube-api-access-zn6gk\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:42.036641 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.036637 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-dshm\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:42.036641 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.036646 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tmp-dir\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:42.036857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.036654 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e6ebbf-766b-47a2-972a-0adae9f6ccda-tls-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:42.095638 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.095615 2573 generic.go:356] "Generic (PLEG): container finished" podID="57e6ebbf-766b-47a2-972a-0adae9f6ccda" containerID="443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9" exitCode=0 Sep 11 15:33:42.095822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.095659 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" Sep 11 15:33:42.095822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.095707 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" event={"ID":"57e6ebbf-766b-47a2-972a-0adae9f6ccda","Type":"ContainerDied","Data":"443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9"} Sep 11 15:33:42.095822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.095746 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw" event={"ID":"57e6ebbf-766b-47a2-972a-0adae9f6ccda","Type":"ContainerDied","Data":"b366307cf8364ae288dd2babd9a6792c89853590b8e4edd8c0669d74599be556"} Sep 11 15:33:42.095822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.095763 2573 scope.go:117] "RemoveContainer" containerID="443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9" Sep 11 15:33:42.105565 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.105549 2573 scope.go:117] "RemoveContainer" containerID="443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9" Sep 11 15:33:42.105876 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:33:42.105857 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9\": container with ID starting with 443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9 not found: ID does not exist" containerID="443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9" Sep 11 15:33:42.105934 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.105885 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9"} err="failed to get container status \"443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9\": rpc error: code = NotFound desc = could not find container \"443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9\": container with ID starting with 443f8ef6a58ba524d9b94089d36ee3f97a5397a70b1e85c43a9e9d555c7569b9 not found: ID does not exist" Sep 11 15:33:42.118469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.118447 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw"] Sep 11 15:33:42.122370 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.122350 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-397e40/rate-test-model-397e40-kserve-5548cb49c5-qvwbw"] Sep 11 15:33:42.493676 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:42.493643 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e6ebbf-766b-47a2-972a-0adae9f6ccda" path="/var/lib/kubelet/pods/57e6ebbf-766b-47a2-972a-0adae9f6ccda/volumes" Sep 11 15:33:45.283855 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.283824 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b"] Sep 11 15:33:45.285852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.285831 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ee57f-29819010-q5z9b"] Sep 11 15:33:45.373647 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.373619 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp"] Sep 11 15:33:45.373902 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.373851 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" podUID="f6a17136-c621-44d1-a042-5736ef91122e" containerName="payload-pre-processing" containerID="cri-o://4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed" gracePeriod=30 Sep 11 15:33:45.553992 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.553949 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28"] Sep 11 15:33:45.558453 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.558425 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ee57f-gmz28"] Sep 11 15:33:45.618475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.618451 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:33:45.664911 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.664882 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvnxj\" (UniqueName: \"kubernetes.io/projected/f6a17136-c621-44d1-a042-5736ef91122e-kube-api-access-kvnxj\") pod \"f6a17136-c621-44d1-a042-5736ef91122e\" (UID: \"f6a17136-c621-44d1-a042-5736ef91122e\") " Sep 11 15:33:45.665024 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.664958 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a17136-c621-44d1-a042-5736ef91122e-plugins-config-volume\") pod \"f6a17136-c621-44d1-a042-5736ef91122e\" (UID: \"f6a17136-c621-44d1-a042-5736ef91122e\") " Sep 11 15:33:45.665327 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.665308 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a17136-c621-44d1-a042-5736ef91122e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6a17136-c621-44d1-a042-5736ef91122e" (UID: "f6a17136-c621-44d1-a042-5736ef91122e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:33:45.666998 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.666975 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a17136-c621-44d1-a042-5736ef91122e-kube-api-access-kvnxj" (OuterVolumeSpecName: "kube-api-access-kvnxj") pod "f6a17136-c621-44d1-a042-5736ef91122e" (UID: "f6a17136-c621-44d1-a042-5736ef91122e"). InnerVolumeSpecName "kube-api-access-kvnxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:45.766471 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.766443 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a17136-c621-44d1-a042-5736ef91122e-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:45.766471 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:45.766468 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvnxj\" (UniqueName: \"kubernetes.io/projected/f6a17136-c621-44d1-a042-5736ef91122e-kube-api-access-kvnxj\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:33:46.121335 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.121310 2573 generic.go:356] "Generic (PLEG): container finished" podID="f6a17136-c621-44d1-a042-5736ef91122e" containerID="4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed" exitCode=0 Sep 11 15:33:46.121475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.121354 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" Sep 11 15:33:46.121475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.121406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" event={"ID":"f6a17136-c621-44d1-a042-5736ef91122e","Type":"ContainerDied","Data":"4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed"} Sep 11 15:33:46.121475 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.121460 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp" event={"ID":"f6a17136-c621-44d1-a042-5736ef91122e","Type":"ContainerDied","Data":"608736ee21f6783b50dffdb319b1608d887535751e3afb531d6d659c209f1610"} Sep 11 15:33:46.121598 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.121488 2573 scope.go:117] "RemoveContainer" containerID="4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed" Sep 11 15:33:46.130647 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.130631 2573 scope.go:117] "RemoveContainer" containerID="4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed" Sep 11 15:33:46.130912 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:33:46.130894 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed\": container with ID starting with 4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed not found: ID does not exist" containerID="4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed" Sep 11 15:33:46.130985 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.130918 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed"} err="failed to get container status \"4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed\": rpc error: code = NotFound desc = could not find container \"4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed\": container with ID starting with 4876d7c1d5cfe582e3dac4e1e0d2a101763760c06a02cc780b2f8dbc19aa98ed not found: ID does not exist" Sep 11 15:33:46.142832 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.142762 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp"] Sep 11 15:33:46.148315 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.148224 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ee57f-6f5dcc7d6-4gzcp"] Sep 11 15:33:46.492798 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.492719 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67400989-7226-4ee7-bf8b-a771fd5a852f" path="/var/lib/kubelet/pods/67400989-7226-4ee7-bf8b-a771fd5a852f/volumes" Sep 11 15:33:46.493107 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.493010 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e2724c5-49b0-4d33-bd44-9ef32af6fd9c" path="/var/lib/kubelet/pods/7e2724c5-49b0-4d33-bd44-9ef32af6fd9c/volumes" Sep 11 15:33:46.493299 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:46.493275 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a17136-c621-44d1-a042-5736ef91122e" path="/var/lib/kubelet/pods/f6a17136-c621-44d1-a042-5736ef91122e/volumes" Sep 11 15:33:49.069951 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:49.069918 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:33:49.070921 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:49.070902 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:33:59.906832 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.906753 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9"] Sep 11 15:33:59.907255 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.907150 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e6ebbf-766b-47a2-972a-0adae9f6ccda" containerName="main" Sep 11 15:33:59.907255 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.907163 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e6ebbf-766b-47a2-972a-0adae9f6ccda" containerName="main" Sep 11 15:33:59.907255 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.907178 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a17136-c621-44d1-a042-5736ef91122e" containerName="payload-pre-processing" Sep 11 15:33:59.907255 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.907183 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a17136-c621-44d1-a042-5736ef91122e" containerName="payload-pre-processing" Sep 11 15:33:59.907255 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.907197 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67400989-7226-4ee7-bf8b-a771fd5a852f" containerName="revoke-keys" Sep 11 15:33:59.907255 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.907202 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67400989-7226-4ee7-bf8b-a771fd5a852f" containerName="revoke-keys" Sep 11 15:33:59.907451 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.907269 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a17136-c621-44d1-a042-5736ef91122e" containerName="payload-pre-processing" Sep 11 15:33:59.907451 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.907277 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e6ebbf-766b-47a2-972a-0adae9f6ccda" containerName="main" Sep 11 15:33:59.907451 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.907282 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67400989-7226-4ee7-bf8b-a771fd5a852f" containerName="revoke-keys" Sep 11 15:33:59.911723 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.911702 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:33:59.917944 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.917916 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0ce6e9\"" Sep 11 15:33:59.918100 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.917974 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0ce6e9-dockercfg-dx766\"" Sep 11 15:33:59.919650 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.919625 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9"] Sep 11 15:33:59.989002 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.988976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwkqk\" (UniqueName: \"kubernetes.io/projected/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-kube-api-access-hwkqk\") pod \"payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9\" (UID: \"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:33:59.989108 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:33:59.989049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9\" (UID: \"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:34:00.089676 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:00.089647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwkqk\" (UniqueName: \"kubernetes.io/projected/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-kube-api-access-hwkqk\") pod \"payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9\" (UID: \"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:34:00.089801 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:00.089725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9\" (UID: \"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:34:00.090414 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:00.090394 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9\" (UID: \"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:34:00.097019 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:00.096999 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwkqk\" (UniqueName: \"kubernetes.io/projected/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-kube-api-access-hwkqk\") pod \"payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9\" (UID: \"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:34:00.223742 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:00.223678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:34:00.495520 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:00.495495 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9"] Sep 11 15:34:00.497939 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:34:00.497914 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa8bbe1_84f6_4599_b74e_bc27f7e34f61.slice/crio-3d0a3781fc7eda0ce7f9ae07322582319f5318d2a74c78de3199ca37f36a310a WatchSource:0}: Error finding container 3d0a3781fc7eda0ce7f9ae07322582319f5318d2a74c78de3199ca37f36a310a: Status 404 returned error can't find the container with id 3d0a3781fc7eda0ce7f9ae07322582319f5318d2a74c78de3199ca37f36a310a Sep 11 15:34:01.207047 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:01.207018 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" event={"ID":"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61","Type":"ContainerStarted","Data":"b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119"} Sep 11 15:34:01.207418 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:01.207069 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" event={"ID":"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61","Type":"ContainerStarted","Data":"3d0a3781fc7eda0ce7f9ae07322582319f5318d2a74c78de3199ca37f36a310a"} Sep 11 15:34:01.207418 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:01.207095 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:34:01.222757 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:01.222703 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" podStartSLOduration=2.222689085 podStartE2EDuration="2.222689085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:34:01.221991146 +0000 UTC m=+1523.267316127" watchObservedRunningTime="2026-09-11 15:34:01.222689085 +0000 UTC m=+1523.268014065" Sep 11 15:34:12.214582 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:12.214547 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:34:26.794615 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:26.794563 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9_7aa8bbe1-84f6-4599-b74e-bc27f7e34f61/payload-processing/0.log" Sep 11 15:34:59.288690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:59.288658 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr_22d277cb-ce76-4d1b-bc15-a1f64a898f14/payload-processing/0.log" Sep 11 15:34:59.517550 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:34:59.517521 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9_7aa8bbe1-84f6-4599-b74e-bc27f7e34f61/payload-processing/0.log" Sep 11 15:35:06.493245 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.493219 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7"] Sep 11 15:35:06.496993 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.496975 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:06.499284 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.499263 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f58c81\"" Sep 11 15:35:06.499358 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.499312 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f58c81-dockercfg-f4jnv\"" Sep 11 15:35:06.505693 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.505669 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7"] Sep 11 15:35:06.609060 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.609034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80d3cf1-e490-40ee-b22e-47c934de1988-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7\" (UID: \"b80d3cf1-e490-40ee-b22e-47c934de1988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:06.609222 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.609091 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pcct\" (UniqueName: \"kubernetes.io/projected/b80d3cf1-e490-40ee-b22e-47c934de1988-kube-api-access-6pcct\") pod \"payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7\" (UID: \"b80d3cf1-e490-40ee-b22e-47c934de1988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:06.710356 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.710328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcct\" (UniqueName: \"kubernetes.io/projected/b80d3cf1-e490-40ee-b22e-47c934de1988-kube-api-access-6pcct\") pod \"payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7\" (UID: \"b80d3cf1-e490-40ee-b22e-47c934de1988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:06.710500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.710399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80d3cf1-e490-40ee-b22e-47c934de1988-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7\" (UID: \"b80d3cf1-e490-40ee-b22e-47c934de1988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:06.711055 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.711036 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80d3cf1-e490-40ee-b22e-47c934de1988-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7\" (UID: \"b80d3cf1-e490-40ee-b22e-47c934de1988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:06.719120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.719103 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pcct\" (UniqueName: \"kubernetes.io/projected/b80d3cf1-e490-40ee-b22e-47c934de1988-kube-api-access-6pcct\") pod \"payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7\" (UID: \"b80d3cf1-e490-40ee-b22e-47c934de1988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:06.808382 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:06.808330 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:07.079180 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:07.079150 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7"] Sep 11 15:35:07.082491 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:35:07.082458 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80d3cf1_e490_40ee_b22e_47c934de1988.slice/crio-b8bcb4b3211458d85b0700039523d435aa860615884f38d305cc42733ad5d826 WatchSource:0}: Error finding container b8bcb4b3211458d85b0700039523d435aa860615884f38d305cc42733ad5d826: Status 404 returned error can't find the container with id b8bcb4b3211458d85b0700039523d435aa860615884f38d305cc42733ad5d826 Sep 11 15:35:07.560889 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:07.560806 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" event={"ID":"b80d3cf1-e490-40ee-b22e-47c934de1988","Type":"ContainerStarted","Data":"2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70"} Sep 11 15:35:07.560889 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:07.560868 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" event={"ID":"b80d3cf1-e490-40ee-b22e-47c934de1988","Type":"ContainerStarted","Data":"b8bcb4b3211458d85b0700039523d435aa860615884f38d305cc42733ad5d826"} Sep 11 15:35:07.561283 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:07.561006 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:07.576159 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:07.576108 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" podStartSLOduration=1.576086944 podStartE2EDuration="1.576086944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:35:07.574759651 +0000 UTC m=+1589.620084634" watchObservedRunningTime="2026-09-11 15:35:07.576086944 +0000 UTC m=+1589.621411924" Sep 11 15:35:18.567325 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:18.567295 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:33.644501 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:33.644425 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s"] Sep 11 15:35:33.648027 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:33.648009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" Sep 11 15:35:33.650337 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:33.650313 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:35:33.650469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:33.650357 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:35:33.654635 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:33.654608 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s"] Sep 11 15:35:33.721469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:33.721441 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19684632-7721-4e2b-ac04-7ff2355a8cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s\" (UID: \"19684632-7721-4e2b-ac04-7ff2355a8cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" Sep 11 15:35:33.822188 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:33.822150 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19684632-7721-4e2b-ac04-7ff2355a8cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s\" (UID: \"19684632-7721-4e2b-ac04-7ff2355a8cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" Sep 11 15:35:33.822742 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:33.822723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19684632-7721-4e2b-ac04-7ff2355a8cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s\" (UID: \"19684632-7721-4e2b-ac04-7ff2355a8cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" Sep 11 15:35:33.961651 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:33.961586 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" Sep 11 15:35:34.231508 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:34.231311 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s"] Sep 11 15:35:34.234333 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:35:34.234297 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19684632_7721_4e2b_ac04_7ff2355a8cad.slice/crio-cded98ac719da78ba9d33ccae6e6c39c5acb1b3c0dd47a958485c5ce53cee9b2 WatchSource:0}: Error finding container cded98ac719da78ba9d33ccae6e6c39c5acb1b3c0dd47a958485c5ce53cee9b2: Status 404 returned error can't find the container with id cded98ac719da78ba9d33ccae6e6c39c5acb1b3c0dd47a958485c5ce53cee9b2 Sep 11 15:35:34.703206 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:34.703161 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" event={"ID":"19684632-7721-4e2b-ac04-7ff2355a8cad","Type":"ContainerStarted","Data":"5f6a302f24cacdcd0e44c8434c0e0de6bd2464b1fdaddfe9534835ce15f72fdc"} Sep 11 15:35:34.703584 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:34.703226 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" event={"ID":"19684632-7721-4e2b-ac04-7ff2355a8cad","Type":"ContainerStarted","Data":"cded98ac719da78ba9d33ccae6e6c39c5acb1b3c0dd47a958485c5ce53cee9b2"} Sep 11 15:35:34.720424 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:34.720377 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" podStartSLOduration=1.720362208 podStartE2EDuration="1.720362208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:35:34.718279697 +0000 UTC m=+1616.763604676" watchObservedRunningTime="2026-09-11 15:35:34.720362208 +0000 UTC m=+1616.765687225" Sep 11 15:35:35.710545 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:35.710511 2573 generic.go:356] "Generic (PLEG): container finished" podID="19684632-7721-4e2b-ac04-7ff2355a8cad" containerID="5f6a302f24cacdcd0e44c8434c0e0de6bd2464b1fdaddfe9534835ce15f72fdc" exitCode=0 Sep 11 15:35:35.710953 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:35.710605 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" event={"ID":"19684632-7721-4e2b-ac04-7ff2355a8cad","Type":"ContainerDied","Data":"5f6a302f24cacdcd0e44c8434c0e0de6bd2464b1fdaddfe9534835ce15f72fdc"} Sep 11 15:35:36.842421 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:36.842398 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" Sep 11 15:35:36.947415 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:36.947391 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19684632-7721-4e2b-ac04-7ff2355a8cad-maas-api-service-ca\") pod \"19684632-7721-4e2b-ac04-7ff2355a8cad\" (UID: \"19684632-7721-4e2b-ac04-7ff2355a8cad\") " Sep 11 15:35:36.947713 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:36.947692 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19684632-7721-4e2b-ac04-7ff2355a8cad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19684632-7721-4e2b-ac04-7ff2355a8cad" (UID: "19684632-7721-4e2b-ac04-7ff2355a8cad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:35:37.048890 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:37.048833 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19684632-7721-4e2b-ac04-7ff2355a8cad-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:35:37.723561 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:37.723532 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" Sep 11 15:35:37.723764 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:37.723567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s" event={"ID":"19684632-7721-4e2b-ac04-7ff2355a8cad","Type":"ContainerDied","Data":"cded98ac719da78ba9d33ccae6e6c39c5acb1b3c0dd47a958485c5ce53cee9b2"} Sep 11 15:35:37.723764 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:37.723609 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cded98ac719da78ba9d33ccae6e6c39c5acb1b3c0dd47a958485c5ce53cee9b2" Sep 11 15:35:38.835088 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:38.835054 2573 scope.go:117] "RemoveContainer" containerID="4f87dc106edb793049a23315a18312eb631d2e99299535aef6f6dc1e24278bdc" Sep 11 15:35:44.746141 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:44.746043 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7"] Sep 11 15:35:44.746569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:44.746325 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" podUID="b80d3cf1-e490-40ee-b22e-47c934de1988" containerName="payload-pre-processing" containerID="cri-o://2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70" gracePeriod=30 Sep 11 15:35:44.943616 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:44.943586 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s"] Sep 11 15:35:44.946738 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:44.946691 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f58c81-x457s"] Sep 11 15:35:45.018224 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.018202 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:45.118464 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.118442 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80d3cf1-e490-40ee-b22e-47c934de1988-plugins-config-volume\") pod \"b80d3cf1-e490-40ee-b22e-47c934de1988\" (UID: \"b80d3cf1-e490-40ee-b22e-47c934de1988\") " Sep 11 15:35:45.118603 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.118483 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pcct\" (UniqueName: \"kubernetes.io/projected/b80d3cf1-e490-40ee-b22e-47c934de1988-kube-api-access-6pcct\") pod \"b80d3cf1-e490-40ee-b22e-47c934de1988\" (UID: \"b80d3cf1-e490-40ee-b22e-47c934de1988\") " Sep 11 15:35:45.118826 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.118797 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80d3cf1-e490-40ee-b22e-47c934de1988-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b80d3cf1-e490-40ee-b22e-47c934de1988" (UID: "b80d3cf1-e490-40ee-b22e-47c934de1988"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:35:45.120711 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.120693 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80d3cf1-e490-40ee-b22e-47c934de1988-kube-api-access-6pcct" (OuterVolumeSpecName: "kube-api-access-6pcct") pod "b80d3cf1-e490-40ee-b22e-47c934de1988" (UID: "b80d3cf1-e490-40ee-b22e-47c934de1988"). InnerVolumeSpecName "kube-api-access-6pcct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:35:45.219485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.219461 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80d3cf1-e490-40ee-b22e-47c934de1988-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:35:45.219485 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.219483 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pcct\" (UniqueName: \"kubernetes.io/projected/b80d3cf1-e490-40ee-b22e-47c934de1988-kube-api-access-6pcct\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:35:45.768856 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.768816 2573 generic.go:356] "Generic (PLEG): container finished" podID="b80d3cf1-e490-40ee-b22e-47c934de1988" containerID="2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70" exitCode=0 Sep 11 15:35:45.769377 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.768864 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" Sep 11 15:35:45.769377 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.768907 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" event={"ID":"b80d3cf1-e490-40ee-b22e-47c934de1988","Type":"ContainerDied","Data":"2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70"} Sep 11 15:35:45.769377 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.768970 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7" event={"ID":"b80d3cf1-e490-40ee-b22e-47c934de1988","Type":"ContainerDied","Data":"b8bcb4b3211458d85b0700039523d435aa860615884f38d305cc42733ad5d826"} Sep 11 15:35:45.769377 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.769006 2573 scope.go:117] "RemoveContainer" containerID="2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70" Sep 11 15:35:45.779872 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.779852 2573 scope.go:117] "RemoveContainer" containerID="2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70" Sep 11 15:35:45.780159 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:35:45.780139 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70\": container with ID starting with 2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70 not found: ID does not exist" containerID="2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70" Sep 11 15:35:45.780222 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.780167 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70"} err="failed to get container status \"2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70\": rpc error: code = NotFound desc = could not find container \"2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70\": container with ID starting with 2a521d2502a5d45b87eec129fc4e63afec6db55e7862dd2dd19e7a073c2a3d70 not found: ID does not exist" Sep 11 15:35:45.789533 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.789509 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7"] Sep 11 15:35:45.793269 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:45.793247 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f58c81-5f5c45f486-57hr7"] Sep 11 15:35:46.492822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:46.492759 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19684632-7721-4e2b-ac04-7ff2355a8cad" path="/var/lib/kubelet/pods/19684632-7721-4e2b-ac04-7ff2355a8cad/volumes" Sep 11 15:35:46.493312 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:35:46.493289 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b80d3cf1-e490-40ee-b22e-47c934de1988" path="/var/lib/kubelet/pods/b80d3cf1-e490-40ee-b22e-47c934de1988/volumes" Sep 11 15:36:10.631519 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.631094 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr"] Sep 11 15:36:10.631519 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.631343 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" podUID="22d277cb-ce76-4d1b-bc15-a1f64a898f14" containerName="payload-processing" containerID="cri-o://8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880" gracePeriod=30 Sep 11 15:36:10.638661 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.638632 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n"] Sep 11 15:36:10.638987 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.638880 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" podUID="8759f770-68a2-4bd2-a561-bf4992564118" containerName="payload-pre-processing" containerID="cri-o://04544db77cd9ca71e1c0b607d89c0524a334be79884b57ac55f8825dbc02d410" gracePeriod=30 Sep 11 15:36:10.910798 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.910752 2573 generic.go:356] "Generic (PLEG): container finished" podID="8759f770-68a2-4bd2-a561-bf4992564118" containerID="04544db77cd9ca71e1c0b607d89c0524a334be79884b57ac55f8825dbc02d410" exitCode=0 Sep 11 15:36:10.911004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.910822 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" event={"ID":"8759f770-68a2-4bd2-a561-bf4992564118","Type":"ContainerDied","Data":"04544db77cd9ca71e1c0b607d89c0524a334be79884b57ac55f8825dbc02d410"} Sep 11 15:36:10.911004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.910883 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" event={"ID":"8759f770-68a2-4bd2-a561-bf4992564118","Type":"ContainerDied","Data":"af4bc98bafe46ed31363d2554fdaff0eb01c1c09c7a5d8966d8a29948bda03fb"} Sep 11 15:36:10.911004 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.910905 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af4bc98bafe46ed31363d2554fdaff0eb01c1c09c7a5d8966d8a29948bda03fb" Sep 11 15:36:10.911154 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.911115 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:36:10.912245 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.912214 2573 generic.go:356] "Generic (PLEG): container finished" podID="22d277cb-ce76-4d1b-bc15-a1f64a898f14" containerID="8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880" exitCode=0 Sep 11 15:36:10.912346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.912248 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" event={"ID":"22d277cb-ce76-4d1b-bc15-a1f64a898f14","Type":"ContainerDied","Data":"8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880"} Sep 11 15:36:10.912346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.912298 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" event={"ID":"22d277cb-ce76-4d1b-bc15-a1f64a898f14","Type":"ContainerDied","Data":"2bd41d5f89aea4da0e7b22e0a3fc833f1a400d9f9b5a6517107e6d28fba76433"} Sep 11 15:36:10.912346 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.912334 2573 scope.go:117] "RemoveContainer" containerID="8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880" Sep 11 15:36:10.914543 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.914527 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:36:10.921879 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.921851 2573 scope.go:117] "RemoveContainer" containerID="8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880" Sep 11 15:36:10.922227 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:36:10.922186 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880\": container with ID starting with 8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880 not found: ID does not exist" containerID="8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880" Sep 11 15:36:10.922227 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:10.922219 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880"} err="failed to get container status \"8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880\": rpc error: code = NotFound desc = could not find container \"8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880\": container with ID starting with 8288c6fd4739c32bb2e03bc3da7427aa7875c61ea13065a73266021c9d880880 not found: ID does not exist" Sep 11 15:36:11.021765 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.021740 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d277cb-ce76-4d1b-bc15-a1f64a898f14-plugins-config-volume\") pod \"22d277cb-ce76-4d1b-bc15-a1f64a898f14\" (UID: \"22d277cb-ce76-4d1b-bc15-a1f64a898f14\") " Sep 11 15:36:11.021915 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.021796 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drl4m\" (UniqueName: \"kubernetes.io/projected/8759f770-68a2-4bd2-a561-bf4992564118-kube-api-access-drl4m\") pod \"8759f770-68a2-4bd2-a561-bf4992564118\" (UID: \"8759f770-68a2-4bd2-a561-bf4992564118\") " Sep 11 15:36:11.021915 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.021860 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8759f770-68a2-4bd2-a561-bf4992564118-plugins-config-volume\") pod \"8759f770-68a2-4bd2-a561-bf4992564118\" (UID: \"8759f770-68a2-4bd2-a561-bf4992564118\") " Sep 11 15:36:11.021915 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.021876 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szw4m\" (UniqueName: \"kubernetes.io/projected/22d277cb-ce76-4d1b-bc15-a1f64a898f14-kube-api-access-szw4m\") pod \"22d277cb-ce76-4d1b-bc15-a1f64a898f14\" (UID: \"22d277cb-ce76-4d1b-bc15-a1f64a898f14\") " Sep 11 15:36:11.022198 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.022167 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d277cb-ce76-4d1b-bc15-a1f64a898f14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22d277cb-ce76-4d1b-bc15-a1f64a898f14" (UID: "22d277cb-ce76-4d1b-bc15-a1f64a898f14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:36:11.022297 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.022242 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8759f770-68a2-4bd2-a561-bf4992564118-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8759f770-68a2-4bd2-a561-bf4992564118" (UID: "8759f770-68a2-4bd2-a561-bf4992564118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:36:11.024069 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.024046 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d277cb-ce76-4d1b-bc15-a1f64a898f14-kube-api-access-szw4m" (OuterVolumeSpecName: "kube-api-access-szw4m") pod "22d277cb-ce76-4d1b-bc15-a1f64a898f14" (UID: "22d277cb-ce76-4d1b-bc15-a1f64a898f14"). InnerVolumeSpecName "kube-api-access-szw4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:36:11.024147 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.024072 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8759f770-68a2-4bd2-a561-bf4992564118-kube-api-access-drl4m" (OuterVolumeSpecName: "kube-api-access-drl4m") pod "8759f770-68a2-4bd2-a561-bf4992564118" (UID: "8759f770-68a2-4bd2-a561-bf4992564118"). InnerVolumeSpecName "kube-api-access-drl4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:36:11.122563 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.122535 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8759f770-68a2-4bd2-a561-bf4992564118-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:36:11.122563 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.122560 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szw4m\" (UniqueName: \"kubernetes.io/projected/22d277cb-ce76-4d1b-bc15-a1f64a898f14-kube-api-access-szw4m\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:36:11.122704 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.122569 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d277cb-ce76-4d1b-bc15-a1f64a898f14-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:36:11.122704 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.122578 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drl4m\" (UniqueName: \"kubernetes.io/projected/8759f770-68a2-4bd2-a561-bf4992564118-kube-api-access-drl4m\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:36:11.919087 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.919062 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr" Sep 11 15:36:11.919506 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.919255 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n" Sep 11 15:36:11.946828 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.946709 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr"] Sep 11 15:36:11.949691 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.949656 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8559ee-8557f6bbdc-94vgr"] Sep 11 15:36:11.956981 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.956959 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n"] Sep 11 15:36:11.960000 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:11.959978 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8559ee-85747b59d6-djt8n"] Sep 11 15:36:12.494331 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:12.494302 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22d277cb-ce76-4d1b-bc15-a1f64a898f14" path="/var/lib/kubelet/pods/22d277cb-ce76-4d1b-bc15-a1f64a898f14/volumes" Sep 11 15:36:12.494653 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:12.494641 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8759f770-68a2-4bd2-a561-bf4992564118" path="/var/lib/kubelet/pods/8759f770-68a2-4bd2-a561-bf4992564118/volumes" Sep 11 15:36:38.855655 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:38.855623 2573 scope.go:117] "RemoveContainer" containerID="771c6a40f4b69d494e9684fdd10a18fc638887d64b72302b19b0875ba624e52b" Sep 11 15:36:38.863749 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:38.863732 2573 scope.go:117] "RemoveContainer" containerID="6a47bd0ea340ee6ddd1c194970eb783757a20b6f741133735bf6ffddb71dd93b" Sep 11 15:36:59.354147 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354104 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb"] Sep 11 15:36:59.354606 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354575 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d277cb-ce76-4d1b-bc15-a1f64a898f14" containerName="payload-processing" Sep 11 15:36:59.354606 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354590 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d277cb-ce76-4d1b-bc15-a1f64a898f14" containerName="payload-processing" Sep 11 15:36:59.354606 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354600 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19684632-7721-4e2b-ac04-7ff2355a8cad" containerName="revoke-keys" Sep 11 15:36:59.354710 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354610 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="19684632-7721-4e2b-ac04-7ff2355a8cad" containerName="revoke-keys" Sep 11 15:36:59.354710 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354624 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8759f770-68a2-4bd2-a561-bf4992564118" containerName="payload-pre-processing" Sep 11 15:36:59.354710 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354631 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8759f770-68a2-4bd2-a561-bf4992564118" containerName="payload-pre-processing" Sep 11 15:36:59.354710 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354644 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b80d3cf1-e490-40ee-b22e-47c934de1988" containerName="payload-pre-processing" Sep 11 15:36:59.354710 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354649 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b80d3cf1-e490-40ee-b22e-47c934de1988" containerName="payload-pre-processing" Sep 11 15:36:59.354922 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354717 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d277cb-ce76-4d1b-bc15-a1f64a898f14" containerName="payload-processing" Sep 11 15:36:59.354922 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354725 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="19684632-7721-4e2b-ac04-7ff2355a8cad" containerName="revoke-keys" Sep 11 15:36:59.354922 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354733 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8759f770-68a2-4bd2-a561-bf4992564118" containerName="payload-pre-processing" Sep 11 15:36:59.354922 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.354738 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b80d3cf1-e490-40ee-b22e-47c934de1988" containerName="payload-pre-processing" Sep 11 15:36:59.357998 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.357979 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb" Sep 11 15:36:59.360131 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.360113 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:36:59.360238 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.360173 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:36:59.364002 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.363979 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb"] Sep 11 15:36:59.409462 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.409432 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb\" (UID: \"baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb" Sep 11 15:36:59.510196 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.510169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb\" (UID: \"baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb" Sep 11 15:36:59.510830 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.510812 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb\" (UID: \"baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb" Sep 11 15:36:59.670806 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.670705 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb" Sep 11 15:36:59.945429 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:36:59.945406 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb"] Sep 11 15:36:59.946942 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:36:59.946916 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf3ad28_9a26_4b99_a3a9_c4cd2453f8e9.slice/crio-c9c5ccade6dd102b4a7d576a739d8ca7bb535561f0418d10c90e70d4c2a29e73 WatchSource:0}: Error finding container c9c5ccade6dd102b4a7d576a739d8ca7bb535561f0418d10c90e70d4c2a29e73: Status 404 returned error can't find the container with id c9c5ccade6dd102b4a7d576a739d8ca7bb535561f0418d10c90e70d4c2a29e73 Sep 11 15:37:00.176085 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:00.176055 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb" event={"ID":"baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9","Type":"ContainerStarted","Data":"c9c5ccade6dd102b4a7d576a739d8ca7bb535561f0418d10c90e70d4c2a29e73"} Sep 11 15:37:01.183729 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:01.183695 2573 generic.go:356] "Generic (PLEG): container finished" podID="baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9" containerID="e28efeecdff71a6c6c5b6952e54d337b9b97d9c2f759f4307499c0f481848197" exitCode=0 Sep 11 15:37:01.184147 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:01.183803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb" event={"ID":"baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9","Type":"ContainerDied","Data":"e28efeecdff71a6c6c5b6952e54d337b9b97d9c2f759f4307499c0f481848197"} Sep 11 15:37:02.314230 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:02.314205 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb" Sep 11 15:37:02.334806 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:02.334762 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9-maas-api-service-ca\") pod \"baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9\" (UID: \"baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9\") " Sep 11 15:37:02.335176 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:02.335145 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9" (UID: "baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:02.335284 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:02.335265 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:03.194505 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:03.194470 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb" Sep 11 15:37:03.194505 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:03.194484 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb" event={"ID":"baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9","Type":"ContainerDied","Data":"c9c5ccade6dd102b4a7d576a739d8ca7bb535561f0418d10c90e70d4c2a29e73"} Sep 11 15:37:03.194699 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:03.194525 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9c5ccade6dd102b4a7d576a739d8ca7bb535561f0418d10c90e70d4c2a29e73" Sep 11 15:37:09.980553 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:09.980522 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9"] Sep 11 15:37:09.981092 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:09.980755 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" podUID="7aa8bbe1-84f6-4599-b74e-bc27f7e34f61" containerName="payload-processing" containerID="cri-o://b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119" gracePeriod=30 Sep 11 15:37:10.142798 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.142698 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb"] Sep 11 15:37:10.145693 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.145662 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce6e9-55rhb"] Sep 11 15:37:10.227994 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.227969 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:37:10.230061 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.230038 2573 generic.go:356] "Generic (PLEG): container finished" podID="7aa8bbe1-84f6-4599-b74e-bc27f7e34f61" containerID="b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119" exitCode=0 Sep 11 15:37:10.230175 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.230084 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" Sep 11 15:37:10.230175 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.230106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" event={"ID":"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61","Type":"ContainerDied","Data":"b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119"} Sep 11 15:37:10.230175 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.230152 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9" event={"ID":"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61","Type":"ContainerDied","Data":"3d0a3781fc7eda0ce7f9ae07322582319f5318d2a74c78de3199ca37f36a310a"} Sep 11 15:37:10.230175 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.230174 2573 scope.go:117] "RemoveContainer" containerID="b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119" Sep 11 15:37:10.238694 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.238678 2573 scope.go:117] "RemoveContainer" containerID="b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119" Sep 11 15:37:10.239009 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:37:10.238991 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119\": container with ID starting with b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119 not found: ID does not exist" containerID="b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119" Sep 11 15:37:10.239071 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.239018 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119"} err="failed to get container status \"b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119\": rpc error: code = NotFound desc = could not find container \"b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119\": container with ID starting with b0a678a05147572c62156360917b9e999ef062c349e0254e9502242e1ec6f119 not found: ID does not exist" Sep 11 15:37:10.301250 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.301224 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwkqk\" (UniqueName: \"kubernetes.io/projected/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-kube-api-access-hwkqk\") pod \"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61\" (UID: \"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61\") " Sep 11 15:37:10.301375 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.301306 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-plugins-config-volume\") pod \"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61\" (UID: \"7aa8bbe1-84f6-4599-b74e-bc27f7e34f61\") " Sep 11 15:37:10.301613 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.301592 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7aa8bbe1-84f6-4599-b74e-bc27f7e34f61" (UID: "7aa8bbe1-84f6-4599-b74e-bc27f7e34f61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:10.301681 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.301669 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:10.303465 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.303436 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-kube-api-access-hwkqk" (OuterVolumeSpecName: "kube-api-access-hwkqk") pod "7aa8bbe1-84f6-4599-b74e-bc27f7e34f61" (UID: "7aa8bbe1-84f6-4599-b74e-bc27f7e34f61"). InnerVolumeSpecName "kube-api-access-hwkqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:10.402969 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.402945 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwkqk\" (UniqueName: \"kubernetes.io/projected/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61-kube-api-access-hwkqk\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:10.499446 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.499386 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9" path="/var/lib/kubelet/pods/baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9/volumes" Sep 11 15:37:10.544941 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.544915 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9"] Sep 11 15:37:10.548723 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:10.548690 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ce6e9-7c845bcfcf-mpbv9"] Sep 11 15:37:12.495806 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:12.495757 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa8bbe1-84f6-4599-b74e-bc27f7e34f61" path="/var/lib/kubelet/pods/7aa8bbe1-84f6-4599-b74e-bc27f7e34f61/volumes" Sep 11 15:37:16.838695 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.838661 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8"] Sep 11 15:37:16.839077 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.839066 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9" containerName="revoke-keys" Sep 11 15:37:16.839124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.839079 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9" containerName="revoke-keys" Sep 11 15:37:16.839124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.839104 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa8bbe1-84f6-4599-b74e-bc27f7e34f61" containerName="payload-processing" Sep 11 15:37:16.839124 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.839110 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa8bbe1-84f6-4599-b74e-bc27f7e34f61" containerName="payload-processing" Sep 11 15:37:16.839223 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.839168 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf3ad28-9a26-4b99-a3a9-c4cd2453f8e9" containerName="revoke-keys" Sep 11 15:37:16.839223 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.839179 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa8bbe1-84f6-4599-b74e-bc27f7e34f61" containerName="payload-processing" Sep 11 15:37:16.843580 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.843562 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8" Sep 11 15:37:16.845687 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.845661 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:37:16.845850 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.845753 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:37:16.849423 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.849398 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8"] Sep 11 15:37:16.956276 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:16.956249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37695733-4faa-4a0e-9450-66bf8cb983f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8\" (UID: \"37695733-4faa-4a0e-9450-66bf8cb983f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8" Sep 11 15:37:17.056779 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:17.056753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37695733-4faa-4a0e-9450-66bf8cb983f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8\" (UID: \"37695733-4faa-4a0e-9450-66bf8cb983f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8" Sep 11 15:37:17.057332 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:17.057313 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37695733-4faa-4a0e-9450-66bf8cb983f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8\" (UID: \"37695733-4faa-4a0e-9450-66bf8cb983f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8" Sep 11 15:37:17.156499 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:17.156412 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8" Sep 11 15:37:17.428836 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:17.428738 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8"] Sep 11 15:37:17.432090 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:37:17.432062 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37695733_4faa_4a0e_9450_66bf8cb983f4.slice/crio-4121b22055438570a14f14fcaed6611360c9b487148b036556560f62bb16aefe WatchSource:0}: Error finding container 4121b22055438570a14f14fcaed6611360c9b487148b036556560f62bb16aefe: Status 404 returned error can't find the container with id 4121b22055438570a14f14fcaed6611360c9b487148b036556560f62bb16aefe Sep 11 15:37:18.273855 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:18.273769 2573 generic.go:356] "Generic (PLEG): container finished" podID="37695733-4faa-4a0e-9450-66bf8cb983f4" containerID="4a0ec7ab44c5fabacf02694f550f152b72bb31291297ef2609d7138512af85e2" exitCode=0 Sep 11 15:37:18.274230 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:18.273864 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8" event={"ID":"37695733-4faa-4a0e-9450-66bf8cb983f4","Type":"ContainerDied","Data":"4a0ec7ab44c5fabacf02694f550f152b72bb31291297ef2609d7138512af85e2"} Sep 11 15:37:18.274230 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:18.273917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8" event={"ID":"37695733-4faa-4a0e-9450-66bf8cb983f4","Type":"ContainerStarted","Data":"4121b22055438570a14f14fcaed6611360c9b487148b036556560f62bb16aefe"} Sep 11 15:37:19.409607 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:19.409582 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8" Sep 11 15:37:19.479202 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:19.479176 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37695733-4faa-4a0e-9450-66bf8cb983f4-maas-api-service-ca\") pod \"37695733-4faa-4a0e-9450-66bf8cb983f4\" (UID: \"37695733-4faa-4a0e-9450-66bf8cb983f4\") " Sep 11 15:37:19.479481 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:19.479460 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37695733-4faa-4a0e-9450-66bf8cb983f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37695733-4faa-4a0e-9450-66bf8cb983f4" (UID: "37695733-4faa-4a0e-9450-66bf8cb983f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:19.579857 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:19.579834 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37695733-4faa-4a0e-9450-66bf8cb983f4-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:20.284389 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:20.284360 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8" Sep 11 15:37:20.284579 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:20.284390 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8" event={"ID":"37695733-4faa-4a0e-9450-66bf8cb983f4","Type":"ContainerDied","Data":"4121b22055438570a14f14fcaed6611360c9b487148b036556560f62bb16aefe"} Sep 11 15:37:20.284579 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:20.284441 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4121b22055438570a14f14fcaed6611360c9b487148b036556560f62bb16aefe" Sep 11 15:37:27.392305 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.392263 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn"] Sep 11 15:37:27.395079 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.395053 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-397e40-29819010-l87tn"] Sep 11 15:37:27.421049 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.421016 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn"] Sep 11 15:37:27.421289 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.421260 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" podUID="1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc" containerName="payload-processing" containerID="cri-o://de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df" gracePeriod=30 Sep 11 15:37:27.424904 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.424877 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx"] Sep 11 15:37:27.425094 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.425068 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" podUID="6eadbec4-d689-4d16-b520-21110b97d007" containerName="payload-pre-processing" containerID="cri-o://f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48" gracePeriod=30 Sep 11 15:37:27.577035 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.577000 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8"] Sep 11 15:37:27.581017 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.580992 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-397e40-z96t8"] Sep 11 15:37:27.708336 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.708312 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:37:27.711117 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.711093 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:37:27.846219 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.846178 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eadbec4-d689-4d16-b520-21110b97d007-plugins-config-volume\") pod \"6eadbec4-d689-4d16-b520-21110b97d007\" (UID: \"6eadbec4-d689-4d16-b520-21110b97d007\") " Sep 11 15:37:27.846391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.846234 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n89k\" (UniqueName: \"kubernetes.io/projected/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-kube-api-access-7n89k\") pod \"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc\" (UID: \"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc\") " Sep 11 15:37:27.846391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.846277 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psvm4\" (UniqueName: \"kubernetes.io/projected/6eadbec4-d689-4d16-b520-21110b97d007-kube-api-access-psvm4\") pod \"6eadbec4-d689-4d16-b520-21110b97d007\" (UID: \"6eadbec4-d689-4d16-b520-21110b97d007\") " Sep 11 15:37:27.846391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.846316 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-plugins-config-volume\") pod \"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc\" (UID: \"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc\") " Sep 11 15:37:27.846708 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.846677 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eadbec4-d689-4d16-b520-21110b97d007-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6eadbec4-d689-4d16-b520-21110b97d007" (UID: "6eadbec4-d689-4d16-b520-21110b97d007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:27.846802 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.846687 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc" (UID: "1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:27.848537 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.848514 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-kube-api-access-7n89k" (OuterVolumeSpecName: "kube-api-access-7n89k") pod "1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc" (UID: "1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc"). InnerVolumeSpecName "kube-api-access-7n89k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:27.848602 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.848538 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eadbec4-d689-4d16-b520-21110b97d007-kube-api-access-psvm4" (OuterVolumeSpecName: "kube-api-access-psvm4") pod "6eadbec4-d689-4d16-b520-21110b97d007" (UID: "6eadbec4-d689-4d16-b520-21110b97d007"). InnerVolumeSpecName "kube-api-access-psvm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:27.947730 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.947674 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:27.947730 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.947696 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eadbec4-d689-4d16-b520-21110b97d007-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:27.947730 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.947707 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n89k\" (UniqueName: \"kubernetes.io/projected/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc-kube-api-access-7n89k\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:27.947730 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:27.947717 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvm4\" (UniqueName: \"kubernetes.io/projected/6eadbec4-d689-4d16-b520-21110b97d007-kube-api-access-psvm4\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:28.328327 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.328300 2573 generic.go:356] "Generic (PLEG): container finished" podID="1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc" containerID="de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df" exitCode=0 Sep 11 15:37:28.328469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.328347 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" Sep 11 15:37:28.328469 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.328394 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" event={"ID":"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc","Type":"ContainerDied","Data":"de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df"} Sep 11 15:37:28.328570 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.328474 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn" event={"ID":"1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc","Type":"ContainerDied","Data":"ad1159a6b62677f52d085c07539e918ae27f07f5b61daa7d309c72847e259029"} Sep 11 15:37:28.328570 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.328503 2573 scope.go:117] "RemoveContainer" containerID="de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df" Sep 11 15:37:28.329952 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.329932 2573 generic.go:356] "Generic (PLEG): container finished" podID="6eadbec4-d689-4d16-b520-21110b97d007" containerID="f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48" exitCode=0 Sep 11 15:37:28.330024 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.329981 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" event={"ID":"6eadbec4-d689-4d16-b520-21110b97d007","Type":"ContainerDied","Data":"f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48"} Sep 11 15:37:28.330024 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.329987 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" Sep 11 15:37:28.330024 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.330015 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx" event={"ID":"6eadbec4-d689-4d16-b520-21110b97d007","Type":"ContainerDied","Data":"bc1be91c02be40eb2eddfacf8dd5278db68439f1fec2e98e323a10a52c29b064"} Sep 11 15:37:28.339850 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.339830 2573 scope.go:117] "RemoveContainer" containerID="de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df" Sep 11 15:37:28.340120 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:37:28.340101 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df\": container with ID starting with de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df not found: ID does not exist" containerID="de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df" Sep 11 15:37:28.340172 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.340130 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df"} err="failed to get container status \"de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df\": rpc error: code = NotFound desc = could not find container \"de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df\": container with ID starting with de6c95f228946084044251d337580aee171cd30f72e4733ebf267dedbda679df not found: ID does not exist" Sep 11 15:37:28.340172 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.340148 2573 scope.go:117] "RemoveContainer" containerID="f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48" Sep 11 15:37:28.348547 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.348530 2573 scope.go:117] "RemoveContainer" containerID="f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48" Sep 11 15:37:28.348834 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:37:28.348815 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48\": container with ID starting with f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48 not found: ID does not exist" containerID="f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48" Sep 11 15:37:28.348912 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.348840 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48"} err="failed to get container status \"f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48\": rpc error: code = NotFound desc = could not find container \"f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48\": container with ID starting with f30f89275901d25344a43ab695823f3859a0872118882465eceec5f7f1e7af48 not found: ID does not exist" Sep 11 15:37:28.354218 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.354156 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn"] Sep 11 15:37:28.356255 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.356237 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-397e40-5d86c7b547-84dbn"] Sep 11 15:37:28.365682 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.365662 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx"] Sep 11 15:37:28.369364 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.369345 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-397e40-6584d4c7d6-g6xzx"] Sep 11 15:37:28.492748 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.492721 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc" path="/var/lib/kubelet/pods/1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc/volumes" Sep 11 15:37:28.493210 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.493190 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37695733-4faa-4a0e-9450-66bf8cb983f4" path="/var/lib/kubelet/pods/37695733-4faa-4a0e-9450-66bf8cb983f4/volumes" Sep 11 15:37:28.493566 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.493548 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eadbec4-d689-4d16-b520-21110b97d007" path="/var/lib/kubelet/pods/6eadbec4-d689-4d16-b520-21110b97d007/volumes" Sep 11 15:37:28.493894 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:28.493874 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbf3f787-2f12-404b-b832-530a4521a9f3" path="/var/lib/kubelet/pods/bbf3f787-2f12-404b-b832-530a4521a9f3/volumes" Sep 11 15:37:34.431206 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.431178 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r"] Sep 11 15:37:34.431590 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.431555 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eadbec4-d689-4d16-b520-21110b97d007" containerName="payload-pre-processing" Sep 11 15:37:34.431590 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.431566 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eadbec4-d689-4d16-b520-21110b97d007" containerName="payload-pre-processing" Sep 11 15:37:34.431590 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.431586 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc" containerName="payload-processing" Sep 11 15:37:34.431590 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.431591 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc" containerName="payload-processing" Sep 11 15:37:34.431727 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.431600 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37695733-4faa-4a0e-9450-66bf8cb983f4" containerName="revoke-keys" Sep 11 15:37:34.431727 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.431606 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37695733-4faa-4a0e-9450-66bf8cb983f4" containerName="revoke-keys" Sep 11 15:37:34.431727 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.431668 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eadbec4-d689-4d16-b520-21110b97d007" containerName="payload-pre-processing" Sep 11 15:37:34.431727 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.431676 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37695733-4faa-4a0e-9450-66bf8cb983f4" containerName="revoke-keys" Sep 11 15:37:34.431727 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.431682 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae4adae-0d05-4a7e-b9f2-b646bc4b1bdc" containerName="payload-processing" Sep 11 15:37:34.436087 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.436070 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" Sep 11 15:37:34.438648 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.438608 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:37:34.439441 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.439421 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:37:34.442076 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.442020 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r"] Sep 11 15:37:34.504391 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.504359 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a78d6e5-e540-4e8c-bd79-d19630ffdc2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r\" (UID: \"0a78d6e5-e540-4e8c-bd79-d19630ffdc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" Sep 11 15:37:34.605564 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.605535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a78d6e5-e540-4e8c-bd79-d19630ffdc2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r\" (UID: \"0a78d6e5-e540-4e8c-bd79-d19630ffdc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" Sep 11 15:37:34.606169 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.606145 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a78d6e5-e540-4e8c-bd79-d19630ffdc2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r\" (UID: \"0a78d6e5-e540-4e8c-bd79-d19630ffdc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" Sep 11 15:37:34.750152 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:34.750080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" Sep 11 15:37:35.018767 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:35.018575 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r"] Sep 11 15:37:35.021553 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:37:35.021526 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a78d6e5_e540_4e8c_bd79_d19630ffdc2c.slice/crio-60482f51f84c9aa8684b1d5e716b64de29b9f92aac38cec8783ead1de35db481 WatchSource:0}: Error finding container 60482f51f84c9aa8684b1d5e716b64de29b9f92aac38cec8783ead1de35db481: Status 404 returned error can't find the container with id 60482f51f84c9aa8684b1d5e716b64de29b9f92aac38cec8783ead1de35db481 Sep 11 15:37:35.372524 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:35.372490 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" event={"ID":"0a78d6e5-e540-4e8c-bd79-d19630ffdc2c","Type":"ContainerStarted","Data":"a7985f6a5db1d03d93c74833c997525ae1c2c013c6c6765f0315d06b9471a319"} Sep 11 15:37:35.372668 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:35.372556 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" event={"ID":"0a78d6e5-e540-4e8c-bd79-d19630ffdc2c","Type":"ContainerStarted","Data":"60482f51f84c9aa8684b1d5e716b64de29b9f92aac38cec8783ead1de35db481"} Sep 11 15:37:35.389639 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:35.389596 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" podStartSLOduration=1.389581673 podStartE2EDuration="1.389581673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:37:35.387962814 +0000 UTC m=+1737.433287790" watchObservedRunningTime="2026-09-11 15:37:35.389581673 +0000 UTC m=+1737.434906653" Sep 11 15:37:36.379507 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:36.379472 2573 generic.go:356] "Generic (PLEG): container finished" podID="0a78d6e5-e540-4e8c-bd79-d19630ffdc2c" containerID="a7985f6a5db1d03d93c74833c997525ae1c2c013c6c6765f0315d06b9471a319" exitCode=0 Sep 11 15:37:36.379915 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:36.379516 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" event={"ID":"0a78d6e5-e540-4e8c-bd79-d19630ffdc2c","Type":"ContainerDied","Data":"a7985f6a5db1d03d93c74833c997525ae1c2c013c6c6765f0315d06b9471a319"} Sep 11 15:37:37.518894 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:37.518873 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" Sep 11 15:37:37.629344 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:37.629320 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a78d6e5-e540-4e8c-bd79-d19630ffdc2c-maas-api-service-ca\") pod \"0a78d6e5-e540-4e8c-bd79-d19630ffdc2c\" (UID: \"0a78d6e5-e540-4e8c-bd79-d19630ffdc2c\") " Sep 11 15:37:37.629696 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:37.629675 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a78d6e5-e540-4e8c-bd79-d19630ffdc2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a78d6e5-e540-4e8c-bd79-d19630ffdc2c" (UID: "0a78d6e5-e540-4e8c-bd79-d19630ffdc2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:37.730569 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:37.730512 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a78d6e5-e540-4e8c-bd79-d19630ffdc2c-maas-api-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:38.391801 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:38.391765 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" Sep 11 15:37:38.391801 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:38.391789 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r" event={"ID":"0a78d6e5-e540-4e8c-bd79-d19630ffdc2c","Type":"ContainerDied","Data":"60482f51f84c9aa8684b1d5e716b64de29b9f92aac38cec8783ead1de35db481"} Sep 11 15:37:38.391985 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:38.391828 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60482f51f84c9aa8684b1d5e716b64de29b9f92aac38cec8783ead1de35db481" Sep 11 15:37:38.900370 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:38.900337 2573 scope.go:117] "RemoveContainer" containerID="1eb3d0c67384b4f91ed645ad15a1e09d038c9570d94f30d4a4fa9b086d7d553d" Sep 11 15:37:44.972766 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:44.972737 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w"] Sep 11 15:37:44.973301 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:44.972976 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" podUID="7006525e-e918-413f-88ff-05a518c5b8e6" containerName="payload-processing" containerID="cri-o://eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c" gracePeriod=30 Sep 11 15:37:44.983429 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:44.983398 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl"] Sep 11 15:37:45.136804 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.136728 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r"] Sep 11 15:37:45.138418 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.138393 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abba31-swl8r"] Sep 11 15:37:45.226874 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.226826 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:37:45.393500 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.393469 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7006525e-e918-413f-88ff-05a518c5b8e6-plugins-config-volume\") pod \"7006525e-e918-413f-88ff-05a518c5b8e6\" (UID: \"7006525e-e918-413f-88ff-05a518c5b8e6\") " Sep 11 15:37:45.393642 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.393547 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9smmd\" (UniqueName: \"kubernetes.io/projected/7006525e-e918-413f-88ff-05a518c5b8e6-kube-api-access-9smmd\") pod \"7006525e-e918-413f-88ff-05a518c5b8e6\" (UID: \"7006525e-e918-413f-88ff-05a518c5b8e6\") " Sep 11 15:37:45.393810 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.393769 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7006525e-e918-413f-88ff-05a518c5b8e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7006525e-e918-413f-88ff-05a518c5b8e6" (UID: "7006525e-e918-413f-88ff-05a518c5b8e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:45.395683 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.395656 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7006525e-e918-413f-88ff-05a518c5b8e6-kube-api-access-9smmd" (OuterVolumeSpecName: "kube-api-access-9smmd") pod "7006525e-e918-413f-88ff-05a518c5b8e6" (UID: "7006525e-e918-413f-88ff-05a518c5b8e6"). InnerVolumeSpecName "kube-api-access-9smmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:45.428454 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.428425 2573 generic.go:356] "Generic (PLEG): container finished" podID="7006525e-e918-413f-88ff-05a518c5b8e6" containerID="eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c" exitCode=0 Sep 11 15:37:45.428563 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.428471 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" Sep 11 15:37:45.428563 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.428518 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" event={"ID":"7006525e-e918-413f-88ff-05a518c5b8e6","Type":"ContainerDied","Data":"eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c"} Sep 11 15:37:45.428676 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.428580 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w" event={"ID":"7006525e-e918-413f-88ff-05a518c5b8e6","Type":"ContainerDied","Data":"34f93ee26ede3f4937591147a052b67ced9302b02f2b890d095a60fa9a1e6291"} Sep 11 15:37:45.428676 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.428606 2573 scope.go:117] "RemoveContainer" containerID="eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c" Sep 11 15:37:45.428814 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.428668 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" podUID="3d083bca-313a-467c-b8ad-30ceab688d08" containerName="payload-pre-processing" containerID="cri-o://35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647" gracePeriod=30 Sep 11 15:37:45.438121 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.438094 2573 scope.go:117] "RemoveContainer" containerID="eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c" Sep 11 15:37:45.438390 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:37:45.438371 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c\": container with ID starting with eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c not found: ID does not exist" containerID="eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c" Sep 11 15:37:45.438449 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.438397 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c"} err="failed to get container status \"eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c\": rpc error: code = NotFound desc = could not find container \"eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c\": container with ID starting with eaaf82039b84e1ca6e42212874f9a4f6807bd29c792d096fc0a41f11941b324c not found: ID does not exist" Sep 11 15:37:45.459960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.459935 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w"] Sep 11 15:37:45.466245 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.466225 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abba31-64c99cf9f7-b9w6w"] Sep 11 15:37:45.494438 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.494387 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7006525e-e918-413f-88ff-05a518c5b8e6-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:45.494438 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.494417 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smmd\" (UniqueName: \"kubernetes.io/projected/7006525e-e918-413f-88ff-05a518c5b8e6-kube-api-access-9smmd\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:45.666159 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.666133 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:37:45.797452 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.797377 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5lwd\" (UniqueName: \"kubernetes.io/projected/3d083bca-313a-467c-b8ad-30ceab688d08-kube-api-access-n5lwd\") pod \"3d083bca-313a-467c-b8ad-30ceab688d08\" (UID: \"3d083bca-313a-467c-b8ad-30ceab688d08\") " Sep 11 15:37:45.797452 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.797442 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d083bca-313a-467c-b8ad-30ceab688d08-plugins-config-volume\") pod \"3d083bca-313a-467c-b8ad-30ceab688d08\" (UID: \"3d083bca-313a-467c-b8ad-30ceab688d08\") " Sep 11 15:37:45.797860 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.797828 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d083bca-313a-467c-b8ad-30ceab688d08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d083bca-313a-467c-b8ad-30ceab688d08" (UID: "3d083bca-313a-467c-b8ad-30ceab688d08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:45.799501 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.799480 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d083bca-313a-467c-b8ad-30ceab688d08-kube-api-access-n5lwd" (OuterVolumeSpecName: "kube-api-access-n5lwd") pod "3d083bca-313a-467c-b8ad-30ceab688d08" (UID: "3d083bca-313a-467c-b8ad-30ceab688d08"). InnerVolumeSpecName "kube-api-access-n5lwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:45.898213 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.898186 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lwd\" (UniqueName: \"kubernetes.io/projected/3d083bca-313a-467c-b8ad-30ceab688d08-kube-api-access-n5lwd\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:45.898213 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:45.898211 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d083bca-313a-467c-b8ad-30ceab688d08-plugins-config-volume\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:37:46.434908 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.434875 2573 generic.go:356] "Generic (PLEG): container finished" podID="3d083bca-313a-467c-b8ad-30ceab688d08" containerID="35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647" exitCode=0 Sep 11 15:37:46.435302 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.434915 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" Sep 11 15:37:46.435302 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.434966 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" event={"ID":"3d083bca-313a-467c-b8ad-30ceab688d08","Type":"ContainerDied","Data":"35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647"} Sep 11 15:37:46.435302 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.435015 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl" event={"ID":"3d083bca-313a-467c-b8ad-30ceab688d08","Type":"ContainerDied","Data":"b02e81a4af4a83e22869d758c545100af337db97ebb955c751b7851cb2606533"} Sep 11 15:37:46.435302 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.435039 2573 scope.go:117] "RemoveContainer" containerID="35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647" Sep 11 15:37:46.443693 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.443678 2573 scope.go:117] "RemoveContainer" containerID="35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647" Sep 11 15:37:46.443965 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:37:46.443948 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647\": container with ID starting with 35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647 not found: ID does not exist" containerID="35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647" Sep 11 15:37:46.444016 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.443974 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647"} err="failed to get container status \"35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647\": rpc error: code = NotFound desc = could not find container \"35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647\": container with ID starting with 35f4840c6581508111b7e8dd782f352bb5c919479ae44f46f6b16ea307b8c647 not found: ID does not exist" Sep 11 15:37:46.455622 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.455600 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl"] Sep 11 15:37:46.461143 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.461122 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abba31-6b6b7ff9d-vgcvl"] Sep 11 15:37:46.492886 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.492867 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a78d6e5-e540-4e8c-bd79-d19630ffdc2c" path="/var/lib/kubelet/pods/0a78d6e5-e540-4e8c-bd79-d19630ffdc2c/volumes" Sep 11 15:37:46.493215 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.493203 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d083bca-313a-467c-b8ad-30ceab688d08" path="/var/lib/kubelet/pods/3d083bca-313a-467c-b8ad-30ceab688d08/volumes" Sep 11 15:37:46.493511 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:37:46.493500 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7006525e-e918-413f-88ff-05a518c5b8e6" path="/var/lib/kubelet/pods/7006525e-e918-413f-88ff-05a518c5b8e6/volumes" Sep 11 15:38:38.542620 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:38:38.542594 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:38:38.551662 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:38:38.551631 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:38:38.934157 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:38:38.934128 2573 scope.go:117] "RemoveContainer" containerID="de8c89d69e1c24f1faebc0b76e95b3b78d47d74133103f3998f29c90bf3abfcc" Sep 11 15:39:38.960690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:39:38.960660 2573 scope.go:117] "RemoveContainer" containerID="04544db77cd9ca71e1c0b607d89c0524a334be79884b57ac55f8825dbc02d410" Sep 11 15:39:38.968591 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:39:38.968571 2573 scope.go:117] "RemoveContainer" containerID="648df85361abdc85fb2bc6e1bc6edb9ed30f6b91ef2adb3c15d39fdc3e69ecc7" Sep 11 15:39:38.975677 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:39:38.975655 2573 scope.go:117] "RemoveContainer" containerID="f10613510a546ffef4afa4e83568ca51c8b80409379012514e3419c4701892e7" Sep 11 15:39:38.982690 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:39:38.982673 2573 scope.go:117] "RemoveContainer" containerID="bfb7b09189abb48c43c98c7309335aa45afbce720e988b6c521d4e298340d616" Sep 11 15:41:39.017985 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:41:39.017918 2573 scope.go:117] "RemoveContainer" containerID="5f6a302f24cacdcd0e44c8434c0e0de6bd2464b1fdaddfe9534835ce15f72fdc" Sep 11 15:42:01.486030 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.485998 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x"] Sep 11 15:42:01.486609 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.486535 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a78d6e5-e540-4e8c-bd79-d19630ffdc2c" containerName="revoke-keys" Sep 11 15:42:01.486609 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.486555 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a78d6e5-e540-4e8c-bd79-d19630ffdc2c" containerName="revoke-keys" Sep 11 15:42:01.486609 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.486570 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7006525e-e918-413f-88ff-05a518c5b8e6" containerName="payload-processing" Sep 11 15:42:01.486609 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.486579 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7006525e-e918-413f-88ff-05a518c5b8e6" containerName="payload-processing" Sep 11 15:42:01.486609 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.486603 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d083bca-313a-467c-b8ad-30ceab688d08" containerName="payload-pre-processing" Sep 11 15:42:01.486609 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.486612 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d083bca-313a-467c-b8ad-30ceab688d08" containerName="payload-pre-processing" Sep 11 15:42:01.486951 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.486692 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d083bca-313a-467c-b8ad-30ceab688d08" containerName="payload-pre-processing" Sep 11 15:42:01.486951 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.486705 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7006525e-e918-413f-88ff-05a518c5b8e6" containerName="payload-processing" Sep 11 15:42:01.486951 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.486719 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a78d6e5-e540-4e8c-bd79-d19630ffdc2c" containerName="revoke-keys" Sep 11 15:42:01.490096 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.490028 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.492499 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.492477 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0cbe456e-kserve-self-signed-certs\"" Sep 11 15:42:01.498858 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.498836 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x"] Sep 11 15:42:01.572987 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.572957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5409f24d-177f-488d-8f65-c122016b6061-tls-certs\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.573112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.573008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-dshm\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.573112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.573034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-home\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.573112 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.573090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-tmp-dir\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.573272 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.573142 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-model-cache\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.573272 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.573176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf8fw\" (UniqueName: \"kubernetes.io/projected/5409f24d-177f-488d-8f65-c122016b6061-kube-api-access-nf8fw\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.674328 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.674301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf8fw\" (UniqueName: \"kubernetes.io/projected/5409f24d-177f-488d-8f65-c122016b6061-kube-api-access-nf8fw\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.674448 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.674341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5409f24d-177f-488d-8f65-c122016b6061-tls-certs\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.674448 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.674369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-dshm\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.674448 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.674384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-home\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.674448 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.674430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-tmp-dir\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.674674 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.674457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-model-cache\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.674864 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.674835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-model-cache\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.674942 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.674862 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-home\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.674942 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.674904 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-tmp-dir\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.676844 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.676820 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-dshm\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.677036 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.677022 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5409f24d-177f-488d-8f65-c122016b6061-tls-certs\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.681822 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.681804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf8fw\" (UniqueName: \"kubernetes.io/projected/5409f24d-177f-488d-8f65-c122016b6061-kube-api-access-nf8fw\") pod \"e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:01.813547 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:01.813496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:02.084918 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:02.084895 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x"] Sep 11 15:42:02.086875 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:42:02.086838 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5409f24d_177f_488d_8f65_c122016b6061.slice/crio-d330b7945571163551f5bae3925bcbd55528265d8d21addd6ff73de5a39fb261 WatchSource:0}: Error finding container d330b7945571163551f5bae3925bcbd55528265d8d21addd6ff73de5a39fb261: Status 404 returned error can't find the container with id d330b7945571163551f5bae3925bcbd55528265d8d21addd6ff73de5a39fb261 Sep 11 15:42:02.689876 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:02.689841 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" event={"ID":"5409f24d-177f-488d-8f65-c122016b6061","Type":"ContainerStarted","Data":"c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d"} Sep 11 15:42:02.690225 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:02.689883 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" event={"ID":"5409f24d-177f-488d-8f65-c122016b6061","Type":"ContainerStarted","Data":"d330b7945571163551f5bae3925bcbd55528265d8d21addd6ff73de5a39fb261"} Sep 11 15:42:02.706747 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:02.706700 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" podStartSLOduration=1.7066832 podStartE2EDuration="1.7066832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:42:02.704923692 +0000 UTC m=+2004.750248673" watchObservedRunningTime="2026-09-11 15:42:02.7066832 +0000 UTC m=+2004.752008179" Sep 11 15:42:11.814489 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:11.814445 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:11.814867 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:11.814509 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:11.826820 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:11.826793 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:12.747579 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:12.747550 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:43.237028 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:43.236988 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x"] Sep 11 15:42:58.256829 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.256751 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" podUID="5409f24d-177f-488d-8f65-c122016b6061" containerName="main" containerID="cri-o://c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d" gracePeriod=45 Sep 11 15:42:58.510807 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.510735 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:58.617415 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617385 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf8fw\" (UniqueName: \"kubernetes.io/projected/5409f24d-177f-488d-8f65-c122016b6061-kube-api-access-nf8fw\") pod \"5409f24d-177f-488d-8f65-c122016b6061\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " Sep 11 15:42:58.617568 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617424 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-tmp-dir\") pod \"5409f24d-177f-488d-8f65-c122016b6061\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " Sep 11 15:42:58.617568 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617460 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5409f24d-177f-488d-8f65-c122016b6061-tls-certs\") pod \"5409f24d-177f-488d-8f65-c122016b6061\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " Sep 11 15:42:58.617568 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617532 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-dshm\") pod \"5409f24d-177f-488d-8f65-c122016b6061\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " Sep 11 15:42:58.617568 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617561 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-home\") pod \"5409f24d-177f-488d-8f65-c122016b6061\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " Sep 11 15:42:58.617798 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617615 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-model-cache\") pod \"5409f24d-177f-488d-8f65-c122016b6061\" (UID: \"5409f24d-177f-488d-8f65-c122016b6061\") " Sep 11 15:42:58.617798 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617635 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5409f24d-177f-488d-8f65-c122016b6061" (UID: "5409f24d-177f-488d-8f65-c122016b6061"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:42:58.617919 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617825 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-home" (OuterVolumeSpecName: "home") pod "5409f24d-177f-488d-8f65-c122016b6061" (UID: "5409f24d-177f-488d-8f65-c122016b6061"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:42:58.617973 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617906 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-model-cache" (OuterVolumeSpecName: "model-cache") pod "5409f24d-177f-488d-8f65-c122016b6061" (UID: "5409f24d-177f-488d-8f65-c122016b6061"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:42:58.617973 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617958 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-tmp-dir\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:42:58.618060 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.617976 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-home\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:42:58.619673 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.619650 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5409f24d-177f-488d-8f65-c122016b6061-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5409f24d-177f-488d-8f65-c122016b6061" (UID: "5409f24d-177f-488d-8f65-c122016b6061"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:42:58.619766 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.619729 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-dshm" (OuterVolumeSpecName: "dshm") pod "5409f24d-177f-488d-8f65-c122016b6061" (UID: "5409f24d-177f-488d-8f65-c122016b6061"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:42:58.619766 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.619765 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5409f24d-177f-488d-8f65-c122016b6061-kube-api-access-nf8fw" (OuterVolumeSpecName: "kube-api-access-nf8fw") pod "5409f24d-177f-488d-8f65-c122016b6061" (UID: "5409f24d-177f-488d-8f65-c122016b6061"). InnerVolumeSpecName "kube-api-access-nf8fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:42:58.719189 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.719163 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:42:58.719189 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.719191 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf8fw\" (UniqueName: \"kubernetes.io/projected/5409f24d-177f-488d-8f65-c122016b6061-kube-api-access-nf8fw\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:42:58.719327 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.719201 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5409f24d-177f-488d-8f65-c122016b6061-tls-certs\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:42:58.719327 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.719210 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5409f24d-177f-488d-8f65-c122016b6061-dshm\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\"" Sep 11 15:42:58.965133 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.965095 2573 generic.go:356] "Generic (PLEG): container finished" podID="5409f24d-177f-488d-8f65-c122016b6061" containerID="c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d" exitCode=0 Sep 11 15:42:58.965299 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.965139 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" Sep 11 15:42:58.965299 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.965177 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" event={"ID":"5409f24d-177f-488d-8f65-c122016b6061","Type":"ContainerDied","Data":"c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d"} Sep 11 15:42:58.965299 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.965219 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x" event={"ID":"5409f24d-177f-488d-8f65-c122016b6061","Type":"ContainerDied","Data":"d330b7945571163551f5bae3925bcbd55528265d8d21addd6ff73de5a39fb261"} Sep 11 15:42:58.965299 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.965264 2573 scope.go:117] "RemoveContainer" containerID="c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d" Sep 11 15:42:58.975378 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.975360 2573 scope.go:117] "RemoveContainer" containerID="c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d" Sep 11 15:42:58.975660 ip-10-0-134-20 kubenswrapper[2573]: E0911 15:42:58.975639 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d\": container with ID starting with c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d not found: ID does not exist" containerID="c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d" Sep 11 15:42:58.975735 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.975671 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d"} err="failed to get container status \"c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d\": rpc error: code = NotFound desc = could not find container \"c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d\": container with ID starting with c8d1e46e648576445fdce6acdab450a76cba27daf07cab3d73d1eed3c2e4ab3d not found: ID does not exist" Sep 11 15:42:58.989099 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.989077 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x"] Sep 11 15:42:58.993868 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:42:58.993849 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0cbe456e-kserve-567f444b9f-mjf5x"] Sep 11 15:43:00.493182 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:00.493101 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5409f24d-177f-488d-8f65-c122016b6061" path="/var/lib/kubelet/pods/5409f24d-177f-488d-8f65-c122016b6061/volumes" Sep 11 15:43:38.578985 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:38.578955 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:43:38.583931 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:38.583909 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:43:39.042575 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:39.042498 2573 scope.go:117] "RemoveContainer" containerID="4a0ec7ab44c5fabacf02694f550f152b72bb31291297ef2609d7138512af85e2" Sep 11 15:43:39.050292 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:39.050272 2573 scope.go:117] "RemoveContainer" containerID="a7985f6a5db1d03d93c74833c997525ae1c2c013c6c6765f0315d06b9471a319" Sep 11 15:43:39.057644 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:39.057627 2573 scope.go:117] "RemoveContainer" containerID="e28efeecdff71a6c6c5b6952e54d337b9b97d9c2f759f4307499c0f481848197" Sep 11 15:43:44.057898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:44.057869 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-87954dbdf-zmdnm_6a5a4890-3732-462d-b1cb-d0b127d59b97/copy-manifests/0.log" Sep 11 15:43:44.064843 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:44.064816 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-87954dbdf-zmdnm_6a5a4890-3732-462d-b1cb-d0b127d59b97/manager/0.log" Sep 11 15:43:46.195377 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:46.195343 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vh8nf_cdd28439-9ae9-4e7b-905b-9c32049e3514/manager/0.log" Sep 11 15:43:46.759587 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:46.759562 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85b8bff76-l9bt5_f03e58a3-53d9-435f-911a-a8c5922efe4d/kube-auth-proxy/0.log" Sep 11 15:43:47.110180 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:47.110157 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x9hrr_ba070303-d713-4a08-87d2-473d485c4c21/payload-pre-processing/0.log" Sep 11 15:43:47.728642 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:47.728611 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-rjljg_0df8022c-5528-4100-a802-22fb65dd6791/main/0.log" Sep 11 15:43:48.058741 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:48.058667 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-56hg5_cce98d31-f0f0-491b-b92d-afe822e568b1/main/0.log" Sep 11 15:43:48.280629 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:48.280601 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6zxpl_8ea3459b-80ac-4374-8df3-f2b610097ad6/main/0.log" Sep 11 15:43:53.648556 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.648517 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c7pwf/must-gather-wjq2q"] Sep 11 15:43:53.649151 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.649093 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5409f24d-177f-488d-8f65-c122016b6061" containerName="main" Sep 11 15:43:53.649151 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.649115 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5409f24d-177f-488d-8f65-c122016b6061" containerName="main" Sep 11 15:43:53.649287 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.649201 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5409f24d-177f-488d-8f65-c122016b6061" containerName="main" Sep 11 15:43:53.652261 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.652239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7pwf/must-gather-wjq2q" Sep 11 15:43:53.655999 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.655978 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c7pwf\"/\"kube-root-ca.crt\"" Sep 11 15:43:53.656099 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.656078 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c7pwf\"/\"default-dockercfg-5nlz8\"" Sep 11 15:43:53.656570 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.656557 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c7pwf\"/\"openshift-service-ca.crt\"" Sep 11 15:43:53.666654 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.666622 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7pwf/must-gather-wjq2q"] Sep 11 15:43:53.732306 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.732285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4553950e-5636-4e64-8c81-881f2f452254-must-gather-output\") pod \"must-gather-wjq2q\" (UID: \"4553950e-5636-4e64-8c81-881f2f452254\") " pod="openshift-must-gather-c7pwf/must-gather-wjq2q" Sep 11 15:43:53.732416 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.732324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhtkx\" (UniqueName: \"kubernetes.io/projected/4553950e-5636-4e64-8c81-881f2f452254-kube-api-access-bhtkx\") pod \"must-gather-wjq2q\" (UID: \"4553950e-5636-4e64-8c81-881f2f452254\") " pod="openshift-must-gather-c7pwf/must-gather-wjq2q" Sep 11 15:43:53.832907 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.832880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4553950e-5636-4e64-8c81-881f2f452254-must-gather-output\") pod \"must-gather-wjq2q\" (UID: \"4553950e-5636-4e64-8c81-881f2f452254\") " pod="openshift-must-gather-c7pwf/must-gather-wjq2q" Sep 11 15:43:53.833009 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.832918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhtkx\" (UniqueName: \"kubernetes.io/projected/4553950e-5636-4e64-8c81-881f2f452254-kube-api-access-bhtkx\") pod \"must-gather-wjq2q\" (UID: \"4553950e-5636-4e64-8c81-881f2f452254\") " pod="openshift-must-gather-c7pwf/must-gather-wjq2q" Sep 11 15:43:53.833191 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.833173 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4553950e-5636-4e64-8c81-881f2f452254-must-gather-output\") pod \"must-gather-wjq2q\" (UID: \"4553950e-5636-4e64-8c81-881f2f452254\") " pod="openshift-must-gather-c7pwf/must-gather-wjq2q" Sep 11 15:43:53.845981 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.845954 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhtkx\" (UniqueName: \"kubernetes.io/projected/4553950e-5636-4e64-8c81-881f2f452254-kube-api-access-bhtkx\") pod \"must-gather-wjq2q\" (UID: \"4553950e-5636-4e64-8c81-881f2f452254\") " pod="openshift-must-gather-c7pwf/must-gather-wjq2q" Sep 11 15:43:53.971439 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:53.971378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7pwf/must-gather-wjq2q" Sep 11 15:43:54.240936 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:54.240912 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7pwf/must-gather-wjq2q"] Sep 11 15:43:54.243184 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:43:54.243154 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4553950e_5636_4e64_8c81_881f2f452254.slice/crio-8e625cf735b91b0f228c5b4dff618c571c449ab841a93238d2b47779465dc8f5 WatchSource:0}: Error finding container 8e625cf735b91b0f228c5b4dff618c571c449ab841a93238d2b47779465dc8f5: Status 404 returned error can't find the container with id 8e625cf735b91b0f228c5b4dff618c571c449ab841a93238d2b47779465dc8f5 Sep 11 15:43:54.245048 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:54.245031 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:43:55.252918 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:55.251980 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7pwf/must-gather-wjq2q" event={"ID":"4553950e-5636-4e64-8c81-881f2f452254","Type":"ContainerStarted","Data":"4a4accee295e0f9acbf15443fb35f324931cddefbcf8bd1e786f17a8d3a7b8a4"} Sep 11 15:43:55.252918 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:55.252037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7pwf/must-gather-wjq2q" event={"ID":"4553950e-5636-4e64-8c81-881f2f452254","Type":"ContainerStarted","Data":"6b657e1dc22652961672e9b1d769dce999740e3cb27cdf51f70f7b822a76bb57"} Sep 11 15:43:55.252918 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:55.252055 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7pwf/must-gather-wjq2q" event={"ID":"4553950e-5636-4e64-8c81-881f2f452254","Type":"ContainerStarted","Data":"8e625cf735b91b0f228c5b4dff618c571c449ab841a93238d2b47779465dc8f5"} Sep 11 15:43:55.270673 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:55.270595 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c7pwf/must-gather-wjq2q" podStartSLOduration=1.612164742 podStartE2EDuration="2.270575138s" totalImagesPullingTime="658.410396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:43:55.26908006 +0000 UTC m=+2117.314405041" watchObservedRunningTime="2026-09-11 15:43:55.270575138 +0000 UTC m=+2117.315900120" Sep 11 15:43:56.485980 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:56.485937 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c6p9x_2f504ee5-37b1-4ba9-8331-92ce973dc316/global-pull-secret-syncer/0.log" Sep 11 15:43:56.645665 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:56.645628 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zpl44_14dd5751-9801-4d2e-939c-0930a1293a25/konnectivity-agent/0.log" Sep 11 15:43:56.691740 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:43:56.691707 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-20.ec2.internal_dbcd3be065f634dd293745cbb66f2c7e/haproxy/0.log" Sep 11 15:44:01.908526 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:01.908463 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vh8nf_cdd28439-9ae9-4e7b-905b-9c32049e3514/manager/0.log" Sep 11 15:44:03.457120 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.457038 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-nbpfm_466e17b5-124c-4a5d-8747-98696ecc7701/cluster-monitoring-operator/0.log" Sep 11 15:44:03.506195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.506167 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-588rj_e7bd37f1-5c38-4f3f-a055-f1d386d22ac9/kube-state-metrics/0.log" Sep 11 15:44:03.526605 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.526575 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-588rj_e7bd37f1-5c38-4f3f-a055-f1d386d22ac9/kube-rbac-proxy-main/0.log" Sep 11 15:44:03.549497 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.549460 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-588rj_e7bd37f1-5c38-4f3f-a055-f1d386d22ac9/kube-rbac-proxy-self/0.log" Sep 11 15:44:03.609572 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.609535 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-ttlfd_9929bde3-5ee9-4f05-8e98-7a871c7355e0/monitoring-plugin/0.log" Sep 11 15:44:03.842154 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.842114 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpzrh_82f879dc-186a-4ca5-89e1-31750bdf3e2b/node-exporter/0.log" Sep 11 15:44:03.870871 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.870544 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpzrh_82f879dc-186a-4ca5-89e1-31750bdf3e2b/kube-rbac-proxy/0.log" Sep 11 15:44:03.896242 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.896209 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpzrh_82f879dc-186a-4ca5-89e1-31750bdf3e2b/init-textfile/0.log" Sep 11 15:44:03.926115 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.926087 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-xktpp_07e3e6e2-57a7-4f51-8859-25519d5dfb4a/kube-rbac-proxy-main/0.log" Sep 11 15:44:03.950766 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.950680 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-xktpp_07e3e6e2-57a7-4f51-8859-25519d5dfb4a/kube-rbac-proxy-self/0.log" Sep 11 15:44:03.974137 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:03.974091 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-xktpp_07e3e6e2-57a7-4f51-8859-25519d5dfb4a/openshift-state-metrics/0.log" Sep 11 15:44:04.034019 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.033718 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d355c750-5d72-4d8c-9b43-62eb49b733bb/prometheus/0.log" Sep 11 15:44:04.063817 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.063792 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d355c750-5d72-4d8c-9b43-62eb49b733bb/config-reloader/0.log" Sep 11 15:44:04.090376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.090341 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d355c750-5d72-4d8c-9b43-62eb49b733bb/thanos-sidecar/0.log" Sep 11 15:44:04.116058 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.116020 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d355c750-5d72-4d8c-9b43-62eb49b733bb/kube-rbac-proxy-web/0.log" Sep 11 15:44:04.144292 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.144264 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d355c750-5d72-4d8c-9b43-62eb49b733bb/kube-rbac-proxy/0.log" Sep 11 15:44:04.168685 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.168657 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d355c750-5d72-4d8c-9b43-62eb49b733bb/kube-rbac-proxy-thanos/0.log" Sep 11 15:44:04.192974 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.192949 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d355c750-5d72-4d8c-9b43-62eb49b733bb/init-config-reloader/0.log" Sep 11 15:44:04.517953 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.517917 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54cb8c4954-rsjln_8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c/thanos-query/0.log" Sep 11 15:44:04.559610 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.559576 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54cb8c4954-rsjln_8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c/kube-rbac-proxy-web/0.log" Sep 11 15:44:04.593749 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.593676 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54cb8c4954-rsjln_8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c/kube-rbac-proxy/0.log" Sep 11 15:44:04.628041 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.628010 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54cb8c4954-rsjln_8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c/prom-label-proxy/0.log" Sep 11 15:44:04.664398 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.664362 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54cb8c4954-rsjln_8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c/kube-rbac-proxy-rules/0.log" Sep 11 15:44:04.705580 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:04.705553 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54cb8c4954-rsjln_8c6c127f-ac4d-40f7-a70c-1e2d947b0b7c/kube-rbac-proxy-metrics/0.log" Sep 11 15:44:06.448710 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.448674 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7"] Sep 11 15:44:06.457436 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.457400 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.462187 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.462153 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7"] Sep 11 15:44:06.542705 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.542662 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/1.log" Sep 11 15:44:06.548463 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.548439 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-cqvgw_43b10749-31c5-435a-bbe8-6245ed5bad1a/console-operator/2.log" Sep 11 15:44:06.574645 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.574350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnlr6\" (UniqueName: \"kubernetes.io/projected/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-kube-api-access-fnlr6\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.574645 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.574399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-podres\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.574645 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.574433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-sys\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.574645 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.574479 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-proc\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.574645 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.574532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-lib-modules\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.676164 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.676121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-sys\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.676352 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.676201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-proc\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.676352 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.676261 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-lib-modules\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.676495 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.676357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnlr6\" (UniqueName: \"kubernetes.io/projected/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-kube-api-access-fnlr6\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.676495 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.676391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-podres\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.676607 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.676555 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-podres\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.676666 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.676619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-sys\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.676728 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.676666 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-proc\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.676805 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.676759 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-lib-modules\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.687755 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.686905 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnlr6\" (UniqueName: \"kubernetes.io/projected/75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0-kube-api-access-fnlr6\") pod \"perf-node-gather-daemonset-vfkz7\" (UID: \"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:06.774291 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:06.774192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:07.071863 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:07.071764 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7"] Sep 11 15:44:07.080422 ip-10-0-134-20 kubenswrapper[2573]: W0911 15:44:07.079862 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod75b4b77c_d30e_4e6b_8756_f4b81c3c3aa0.slice/crio-b417bb703389c186cf67797befcb9e9f3a98a123f5b9c40c8f93c2c49014c2a9 WatchSource:0}: Error finding container b417bb703389c186cf67797befcb9e9f3a98a123f5b9c40c8f93c2c49014c2a9: Status 404 returned error can't find the container with id b417bb703389c186cf67797befcb9e9f3a98a123f5b9c40c8f93c2c49014c2a9 Sep 11 15:44:07.145218 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:07.145188 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cd998cc94-db6l7_fc367530-f4f4-4a1d-9cdd-046e94e2f1d3/console/0.log" Sep 11 15:44:07.369986 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:07.369944 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" event={"ID":"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0","Type":"ContainerStarted","Data":"a61d70e45b203eb30ad7d27bf81d7c385c6892a342cb7feeb1fcafff7ffb171c"} Sep 11 15:44:07.370195 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:07.370013 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" event={"ID":"75b4b77c-d30e-4e6b-8756-f4b81c3c3aa0","Type":"ContainerStarted","Data":"b417bb703389c186cf67797befcb9e9f3a98a123f5b9c40c8f93c2c49014c2a9"} Sep 11 15:44:07.370868 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:07.370844 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:07.389028 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:07.388983 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" podStartSLOduration=1.38896758 podStartE2EDuration="1.38896758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:44:07.386424396 +0000 UTC m=+2129.431749375" watchObservedRunningTime="2026-09-11 15:44:07.38896758 +0000 UTC m=+2129.434292560" Sep 11 15:44:08.951896 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:08.951867 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p94fh_07bd3b2e-2168-40ec-8fcf-6a55bfa7468c/dns/0.log" Sep 11 15:44:08.973898 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:08.973870 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p94fh_07bd3b2e-2168-40ec-8fcf-6a55bfa7468c/kube-rbac-proxy/0.log" Sep 11 15:44:09.093498 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:09.093466 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tjhlf_1813d8d4-743c-4b24-b861-aa702a8f5e5b/dns-node-resolver/0.log" Sep 11 15:44:09.608613 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:09.608589 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78d65bbf48-lstr8_67eef969-04fa-4cf5-a5e6-22b3cb066888/registry/0.log" Sep 11 15:44:09.681480 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:09.681451 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r4zv8_3bcac49b-9179-411c-9dd7-1de6585a7a0f/node-ca/0.log" Sep 11 15:44:11.021353 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:11.021325 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85b8bff76-l9bt5_f03e58a3-53d9-435f-911a-a8c5922efe4d/kube-auth-proxy/0.log" Sep 11 15:44:11.135852 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:11.135816 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x9hrr_ba070303-d713-4a08-87d2-473d485c4c21/payload-pre-processing/0.log" Sep 11 15:44:12.075376 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:12.075349 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-smjz7_522b2527-f838-4d7d-95cf-6f2ce8ff4bd4/serve-healthcheck-canary/0.log" Sep 11 15:44:12.767450 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:12.767418 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xdncs_fd2de11f-c409-4a82-af57-9883c2e14767/kube-rbac-proxy/0.log" Sep 11 15:44:12.789370 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:12.789326 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xdncs_fd2de11f-c409-4a82-af57-9883c2e14767/exporter/0.log" Sep 11 15:44:12.810965 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:12.810942 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xdncs_fd2de11f-c409-4a82-af57-9883c2e14767/extractor/0.log" Sep 11 15:44:14.395175 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:14.395132 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-vfkz7" Sep 11 15:44:15.541407 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:15.541317 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-87954dbdf-zmdnm_6a5a4890-3732-462d-b1cb-d0b127d59b97/manager/0.log" Sep 11 15:44:15.569470 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:15.569448 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-87954dbdf-zmdnm_6a5a4890-3732-462d-b1cb-d0b127d59b97/copy-manifests/0.log" Sep 11 15:44:16.913925 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:16.913893 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68f4698bc6-gxs2s_2ab062ec-1aba-4b5e-9c88-806ba856d0bb/manager/0.log" Sep 11 15:44:17.200747 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:17.200663 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85b8bff76-l9bt5_f03e58a3-53d9-435f-911a-a8c5922efe4d/kube-auth-proxy/0.log" Sep 11 15:44:17.328444 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:17.328417 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x9hrr_ba070303-d713-4a08-87d2-473d485c4c21/payload-pre-processing/0.log" Sep 11 15:44:19.164048 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:19.164005 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85b8bff76-l9bt5_f03e58a3-53d9-435f-911a-a8c5922efe4d/kube-auth-proxy/0.log" Sep 11 15:44:19.251960 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:19.251921 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x9hrr_ba070303-d713-4a08-87d2-473d485c4c21/payload-pre-processing/0.log" Sep 11 15:44:23.906081 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:23.906051 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swbrq_1be2d9af-96fa-402c-b577-e51409bf5fe0/kube-multus-additional-cni-plugins/0.log" Sep 11 15:44:23.928188 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:23.928167 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swbrq_1be2d9af-96fa-402c-b577-e51409bf5fe0/egress-router-binary-copy/0.log" Sep 11 15:44:23.951844 ip-10-0-134-20 kubenswrapper[2573]: I0911 15:44:23.951825 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swbrq_1be2d9af-96fa-402c-b577-e51409bf5fe0/cni-plugins/0.log"