Sep 14 09:43:02.118088 ip-10-0-135-40 systemd[1]: Starting Kubernetes Kubelet... Sep 14 09:43:02.559944 ip-10-0-135-40 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 09:43:02.559944 ip-10-0-135-40 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 14 09:43:02.559944 ip-10-0-135-40 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 09:43:02.559944 ip-10-0-135-40 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 14 09:43:02.559944 ip-10-0-135-40 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 09:43:02.562607 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.562525 2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 14 09:43:02.565701 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565687 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 09:43:02.565701 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565702 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565706 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565710 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565713 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565715 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565720 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565724 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565727 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565731 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565734 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565737 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565740 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565743 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565746 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565749 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565752 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565755 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565758 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565761 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 09:43:02.565763 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565764 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565767 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565770 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565773 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565776 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565779 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565782 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565784 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565787 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565790 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565792 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565795 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565797 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565799 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565815 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565817 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565820 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565823 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565825 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565828 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565831 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 09:43:02.566260 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565833 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565836 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565838 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565841 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565844 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565846 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565849 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565851 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565854 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565856 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565859 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565861 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565864 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565867 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565871 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565874 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565877 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565880 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565882 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565885 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 09:43:02.566778 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565888 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565890 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565893 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565895 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565898 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565901 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565903 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565906 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565908 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565911 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565914 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565917 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565919 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565921 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565924 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565927 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565929 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565932 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565934 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565937 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 09:43:02.567274 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565939 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565942 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565946 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565950 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565952 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565957 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565960 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565963 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565966 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565968 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565971 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565974 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565978 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565980 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565983 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565986 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565988 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.565991 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566656 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 09:43:02.567752 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566662 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566665 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566668 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566670 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566673 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566676 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566679 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566681 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566684 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566687 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566690 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566693 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566696 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566698 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566701 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566704 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566706 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566709 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566713 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566716 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 09:43:02.568217 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566718 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566721 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566724 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566726 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566729 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566731 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566734 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566738 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566742 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566745 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566748 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566751 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566753 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566756 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566760 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566764 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566766 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566769 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566772 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 09:43:02.568837 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566775 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566779 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566783 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566787 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566790 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566793 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566796 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566799 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566816 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566819 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566821 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566824 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566827 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566830 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566833 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566835 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566838 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566840 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566843 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 09:43:02.569349 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566846 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566848 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566851 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566853 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566856 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566858 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566861 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566863 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566866 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566868 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566871 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566874 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566876 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566879 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566883 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566885 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566888 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566891 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566893 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566896 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 09:43:02.569821 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566898 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566901 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566903 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566906 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566909 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566911 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566914 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566917 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566919 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566922 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566924 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566927 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566929 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566932 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566934 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566937 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566939 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566942 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566944 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.566947 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 09:43:02.570319 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567333 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567338 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567341 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567344 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567348 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567350 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567353 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567356 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567359 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567362 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567364 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567367 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567370 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567373 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567375 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567378 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567381 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567383 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567386 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567388 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 09:43:02.570875 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567391 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567394 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567396 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567399 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567401 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567404 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567407 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567409 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567412 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567414 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567416 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567419 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567421 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567424 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567426 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567430 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567432 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567435 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567438 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567440 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 09:43:02.571354 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567443 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567446 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567448 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567451 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567467 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567471 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567474 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567478 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567482 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567486 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567491 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567494 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567497 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567500 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567503 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567505 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567508 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567511 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567515 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 09:43:02.571846 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567518 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567521 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567523 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567526 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567529 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567532 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567534 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567537 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567540 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567542 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567544 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567547 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567552 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567554 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567557 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567559 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567562 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567565 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567567 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567570 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 09:43:02.572314 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567572 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567575 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567577 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567580 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567582 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567585 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567587 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567590 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567592 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567595 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567597 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567600 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567602 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567605 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567608 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567610 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567613 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567615 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567618 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.567620 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 09:43:02.572815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567686 2550 flags.go:64] FLAG: --address="0.0.0.0" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567724 2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567733 2550 flags.go:64] FLAG: --anonymous-auth="true" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567737 2550 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567746 2550 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567749 2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567753 2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567757 2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567760 2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567763 2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567767 2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567770 2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567774 2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567777 2550 flags.go:64] FLAG: --cgroup-root="" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567779 2550 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567782 2550 flags.go:64] FLAG: --client-ca-file="" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567785 2550 flags.go:64] FLAG: --cloud-provider="external" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567788 2550 flags.go:64] FLAG: --cluster-dns="[]" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567792 2550 flags.go:64] FLAG: --cluster-domain="" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567795 2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567800 2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567815 2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567818 2550 flags.go:64] FLAG: --container-log-max-files="5" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567822 2550 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 14 09:43:02.573311 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567825 2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567829 2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567832 2550 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567835 2550 flags.go:64] FLAG: --contention-profiling="false" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567838 2550 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567841 2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567844 2550 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567847 2550 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567851 2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567854 2550 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567857 2550 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567860 2550 flags.go:64] FLAG: --enable-load-reader="false" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567865 2550 flags.go:64] FLAG: --enable-server="true" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567868 2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567873 2550 flags.go:64] FLAG: --event-burst="100" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567876 2550 flags.go:64] FLAG: --event-qps="50" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567879 2550 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567882 2550 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567885 2550 flags.go:64] FLAG: --eviction-hard="" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567889 2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567892 2550 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567895 2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567898 2550 flags.go:64] FLAG: --eviction-soft="" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567901 2550 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567904 2550 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 14 09:43:02.573940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567907 2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567911 2550 flags.go:64] FLAG: --experimental-mounter-path="" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567913 2550 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567917 2550 flags.go:64] FLAG: --fail-swap-on="true" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567920 2550 flags.go:64] FLAG: --feature-gates="" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567924 2550 flags.go:64] FLAG: --file-check-frequency="20s" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567927 2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567930 2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567933 2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567936 2550 flags.go:64] FLAG: --healthz-port="10248" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567939 2550 flags.go:64] FLAG: --help="false" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567942 2550 flags.go:64] FLAG: --hostname-override="ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567945 2550 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567948 2550 flags.go:64] FLAG: --http-check-frequency="20s" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567950 2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567954 2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567957 2550 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567960 2550 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567963 2550 flags.go:64] FLAG: --image-service-endpoint="" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567967 2550 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567970 2550 flags.go:64] FLAG: --kube-api-burst="100" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567974 2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567977 2550 flags.go:64] FLAG: --kube-api-qps="50" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567980 2550 flags.go:64] FLAG: --kube-reserved="" Sep 14 09:43:02.574593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567983 2550 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567985 2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567988 2550 flags.go:64] FLAG: --kubelet-cgroups="" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567991 2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567994 2550 flags.go:64] FLAG: --lock-file="" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.567997 2550 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568000 2550 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568003 2550 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568008 2550 flags.go:64] FLAG: --log-json-split-stream="false" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568011 2550 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568014 2550 flags.go:64] FLAG: --log-text-split-stream="false" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568018 2550 flags.go:64] FLAG: --logging-format="text" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568021 2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568024 2550 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568027 2550 flags.go:64] FLAG: --manifest-url="" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568030 2550 flags.go:64] FLAG: --manifest-url-header="" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568034 2550 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568037 2550 flags.go:64] FLAG: --max-open-files="1000000" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568041 2550 flags.go:64] FLAG: --max-pods="110" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568044 2550 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568047 2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568049 2550 flags.go:64] FLAG: --memory-manager-policy="None" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568052 2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568055 2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568058 2550 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 14 09:43:02.575189 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568061 2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568068 2550 flags.go:64] FLAG: --node-status-max-images="50" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568072 2550 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568075 2550 flags.go:64] FLAG: --oom-score-adj="-999" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568078 2550 flags.go:64] FLAG: --pod-cidr="" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568081 2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568086 2550 flags.go:64] FLAG: --pod-manifest-path="" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568089 2550 flags.go:64] FLAG: --pod-max-pids="-1" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568092 2550 flags.go:64] FLAG: --pods-per-core="0" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568095 2550 flags.go:64] FLAG: --port="10250" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568098 2550 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568101 2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-000833ebbf41fb759" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568104 2550 flags.go:64] FLAG: --qos-reserved="" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568107 2550 flags.go:64] FLAG: --read-only-port="10255" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568110 2550 flags.go:64] FLAG: --register-node="true" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568113 2550 flags.go:64] FLAG: --register-with-taints="" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568116 2550 flags.go:64] FLAG: --registry-burst="10" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568119 2550 flags.go:64] FLAG: --registry-qps="5" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568123 2550 flags.go:64] FLAG: --reserved-cpus="" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568126 2550 flags.go:64] FLAG: --reserved-memory="" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568130 2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568133 2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568135 2550 flags.go:64] FLAG: --rotate-certificates="false" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568138 2550 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568141 2550 flags.go:64] FLAG: --runonce="false" Sep 14 09:43:02.576046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568144 2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568147 2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568150 2550 flags.go:64] FLAG: --seccomp-default="false" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568153 2550 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568156 2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568159 2550 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568162 2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568165 2550 flags.go:64] FLAG: --storage-driver-password="root" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568168 2550 flags.go:64] FLAG: --storage-driver-secure="false" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568171 2550 flags.go:64] FLAG: --storage-driver-table="stats" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568175 2550 flags.go:64] FLAG: --storage-driver-user="root" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568178 2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568181 2550 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568184 2550 flags.go:64] FLAG: --system-cgroups="" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568187 2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568192 2550 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568195 2550 flags.go:64] FLAG: --tls-cert-file="" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568197 2550 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568201 2550 flags.go:64] FLAG: --tls-min-version="" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568204 2550 flags.go:64] FLAG: --tls-private-key-file="" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568207 2550 flags.go:64] FLAG: --topology-manager-policy="none" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568209 2550 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568212 2550 flags.go:64] FLAG: --topology-manager-scope="container" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568215 2550 flags.go:64] FLAG: --v="2" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568219 2550 flags.go:64] FLAG: --version="false" Sep 14 09:43:02.577147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568224 2550 flags.go:64] FLAG: --vmodule="" Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568227 2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.568231 2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568338 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568342 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568345 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568349 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568352 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568355 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568357 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568360 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568363 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568366 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568368 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568371 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568373 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568376 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568379 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568382 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568384 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 09:43:02.578236 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568387 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568390 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568393 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568395 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568398 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568400 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568403 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568405 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568408 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568410 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568413 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568415 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568418 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568421 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568423 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568426 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568428 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568431 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568433 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 09:43:02.579136 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568436 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568438 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568441 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568444 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568446 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568449 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568452 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568454 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568457 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568459 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568462 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568465 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568468 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568470 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568473 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568476 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568479 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568482 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568485 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568487 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 09:43:02.579930 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568490 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568492 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568495 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568497 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568500 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568503 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568506 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568508 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568511 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568513 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568516 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568519 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568523 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568526 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568528 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568531 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568533 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568536 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568538 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568541 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 09:43:02.580451 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568544 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568546 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568549 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568553 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568556 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568559 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568562 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568565 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568568 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568571 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568573 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568576 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568579 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568581 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568584 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568586 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568589 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568591 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568594 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 09:43:02.580973 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568596 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568600 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568603 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.568606 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.569270 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.578171 2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.578190 2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.578215 2550 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.578224 2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578314 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578322 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578327 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578332 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578337 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 09:43:02.581449 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578341 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578345 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578349 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578354 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578358 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578362 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578366 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578378 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578383 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578387 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578392 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578396 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578400 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578404 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578411 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578424 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578429 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578433 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578447 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578453 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 09:43:02.582109 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578457 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578461 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578465 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578470 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578475 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578479 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578484 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578488 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578493 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578497 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578501 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578505 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578514 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578518 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578522 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578526 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578530 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578534 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578538 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 09:43:02.582977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578542 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578554 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578559 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578563 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578567 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578571 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578575 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578579 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578586 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578591 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578596 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578600 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578604 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578608 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578612 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578617 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578621 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578626 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578630 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578634 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 09:43:02.583481 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578639 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578643 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578647 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578651 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578656 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578660 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578664 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578668 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578672 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578676 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578680 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578685 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578691 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578697 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578709 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578713 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578718 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578722 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578726 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578730 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 09:43:02.584132 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578734 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578738 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578742 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578747 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578751 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578756 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578760 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578764 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578768 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578772 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578776 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578781 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578785 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578789 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.578794 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 09:43:02.584625 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.578818 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579070 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579078 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579083 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579088 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579093 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579098 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579102 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579107 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579111 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579115 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579119 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579133 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579138 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579142 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579146 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579150 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579154 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579158 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 09:43:02.585076 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579162 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579166 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579170 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579175 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579179 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579183 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579187 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579191 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579196 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579200 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579206 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579212 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579217 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579221 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579225 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579230 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579234 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579238 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579242 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579246 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 09:43:02.585544 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579250 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579254 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579258 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579262 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579267 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579271 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579283 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579287 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579291 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579295 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579299 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579303 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579307 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579311 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579315 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579320 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579324 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579328 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579332 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579336 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 09:43:02.586213 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579340 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579344 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579349 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579353 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579358 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579362 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579369 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579374 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579379 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579383 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579388 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579392 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579396 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579401 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579405 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579409 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579414 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579418 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579422 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 09:43:02.586730 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579433 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579438 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579442 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579446 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579450 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579456 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579461 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579466 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579470 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579473 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579477 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579482 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579486 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579490 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579494 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579498 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579502 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579506 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579510 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 09:43:02.587221 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579514 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 09:43:02.587991 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579518 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 09:43:02.587991 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:02.579522 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 14 09:43:02.587991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.579530 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 09:43:02.587991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.579676 2550 server.go:962] "Client rotation is on, will bootstrap in background" Sep 14 09:43:02.587991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.583688 2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 14 09:43:02.587991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.584820 2550 server.go:1019] "Starting client certificate rotation" Sep 14 09:43:02.587991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.584952 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 09:43:02.587991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.585011 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 09:43:02.588339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.588067 2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 09:43:02.596187 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.596164 2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 09:43:02.607291 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.607269 2550 log.go:25] "Validated CRI v1 runtime API" Sep 14 09:43:02.610716 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.610701 2550 log.go:25] "Validated CRI v1 image API" Sep 14 09:43:02.611580 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.611565 2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 14 09:43:02.614869 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.614848 2550 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 86029990-0152-4d3e-b4fe-d2a46ee0a1ff:/dev/nvme0n1p4 d8d5ebab-36c2-4e14-a7e0-34472bff5874:/dev/nvme0n1p3] Sep 14 09:43:02.614993 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.614867 2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 14 09:43:02.617342 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.617326 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 09:43:02.619896 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.619771 2550 manager.go:216] Machine: {Timestamp:2026-09-14 09:43:02.618469095 +0000 UTC m=+0.382425075 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3055342 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c794ed3b3a0bf0722a4a21f135a91 SystemUUID:ec2c794e-d3b3-a0bf-0722-a4a21f135a91 BootID:f536255d-b1dd-4342-ad27-80004ac86c01 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:db:52:67:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:db:52:67:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:8b:e4:ca:0e:0c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 14 09:43:02.619896 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.619890 2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 14 09:43:02.620014 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.619961 2550 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 14 09:43:02.621835 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.621815 2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 14 09:43:02.621972 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.621838 2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-40.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 14 09:43:02.623452 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.623442 2550 topology_manager.go:138] "Creating topology manager with none policy" Sep 14 09:43:02.623491 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.623455 2550 container_manager_linux.go:309] "Creating device plugin manager" Sep 14 09:43:02.623491 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.623463 2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 09:43:02.624313 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.624302 2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 09:43:02.625636 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.625626 2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 14 09:43:02.629376 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.629365 2550 state_mem.go:36] "Initialized new in-memory state store" Sep 14 09:43:02.629489 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.629479 2550 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 14 09:43:02.629521 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.629508 2550 kubelet.go:486] "Attempting to sync node with API server" Sep 14 09:43:02.629554 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.629518 2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 14 09:43:02.629554 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.629536 2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 14 09:43:02.629554 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.629545 2550 kubelet.go:398] "Adding apiserver pod source" Sep 14 09:43:02.629554 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.629554 2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 14 09:43:02.629929 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.629915 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 09:43:02.629971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.629941 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 09:43:02.632434 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.632409 2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 14 09:43:02.635474 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.635461 2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 14 09:43:02.635514 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.635490 2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 14 09:43:02.637598 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637584 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 14 09:43:02.637662 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637602 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 14 09:43:02.637662 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637609 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 14 09:43:02.637662 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637615 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 14 09:43:02.637662 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637628 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 14 09:43:02.637662 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637636 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 14 09:43:02.637662 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637650 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 14 09:43:02.637662 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637656 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 14 09:43:02.637662 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637664 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 14 09:43:02.637905 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637670 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 14 09:43:02.637905 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637678 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 14 09:43:02.637905 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637683 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 14 09:43:02.637905 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637707 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 14 09:43:02.637905 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.637715 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 14 09:43:02.641483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.641471 2550 server.go:1268] "Started kubelet" Sep 14 09:43:02.641557 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.641537 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 14 09:43:02.642265 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.641606 2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 14 09:43:02.642490 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.642428 2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 14 09:43:02.642586 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.642513 2550 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 14 09:43:02.642586 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.642538 2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-40.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 14 09:43:02.642689 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.642651 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 14 09:43:02.643136 ip-10-0-135-40 systemd[1]: Started Kubernetes Kubelet. Sep 14 09:43:02.644281 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.644064 2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 14 09:43:02.645331 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.645319 2550 server.go:310] "Adding debug handlers to kubelet server" Sep 14 09:43:02.652174 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.652154 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 14 09:43:02.652842 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.652822 2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 14 09:43:02.652940 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.652856 2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 14 09:43:02.652940 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.651738 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-40.ec2.internal.18d526b444e4bea3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-40.ec2.internal,UID:ip-10-0-135-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-40.ec2.internal,},FirstTimestamp:2026-09-14 09:43:02.641450659 +0000 UTC m=+0.405406643,LastTimestamp:2026-09-14 09:43:02.641450659 +0000 UTC m=+0.405406643,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-40.ec2.internal,}" Sep 14 09:43:02.653052 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.652945 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:02.653664 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653647 2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 14 09:43:02.653664 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653651 2550 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 14 09:43:02.653789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653679 2550 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 14 09:43:02.653789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653707 2550 factory.go:153] Registering CRI-O factory Sep 14 09:43:02.653789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653723 2550 factory.go:223] Registration of the crio container factory successfully Sep 14 09:43:02.653789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653752 2550 reconstruct.go:97] "Volume reconstruction finished" Sep 14 09:43:02.653789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653764 2550 reconciler.go:29] "Reconciler: start to sync state" Sep 14 09:43:02.653789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653789 2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 14 09:43:02.654042 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653799 2550 factory.go:55] Registering systemd factory Sep 14 09:43:02.654042 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653821 2550 factory.go:223] Registration of the systemd container factory successfully Sep 14 09:43:02.654042 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653834 2550 factory.go:103] Registering Raw factory Sep 14 09:43:02.654042 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.653843 2550 manager.go:1196] Started watching for new ooms in manager Sep 14 09:43:02.654232 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.654219 2550 manager.go:349] Starting recovery of all containers Sep 14 09:43:02.663500 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.663317 2550 manager.go:354] Recovery completed Sep 14 09:43:02.664385 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.664361 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 14 09:43:02.664479 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.664446 2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 14 09:43:02.668336 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.668324 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 09:43:02.670604 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.670589 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 14 09:43:02.670680 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.670617 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 09:43:02.670680 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.670627 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 14 09:43:02.671330 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.671192 2550 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 14 09:43:02.671330 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.671207 2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 14 09:43:02.671330 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.671214 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 09:43:02.671330 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.671224 2550 state_mem.go:36] "Initialized new in-memory state store" Sep 14 09:43:02.671960 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.671944 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 14 09:43:02.672015 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.671974 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 09:43:02.672015 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.671989 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 14 09:43:02.672427 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.672405 2550 policy_none.go:49] "None policy: Start" Sep 14 09:43:02.672427 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.672424 2550 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 14 09:43:02.672541 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.672439 2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 14 09:43:02.673365 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.673353 2550 policy_none.go:47] "Start" Sep 14 09:43:02.679610 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.679594 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jxnxm" Sep 14 09:43:02.681451 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.681391 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-40.ec2.internal.18d526b446a19ca6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-40.ec2.internal,UID:ip-10-0-135-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-40.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-40.ec2.internal,},FirstTimestamp:2026-09-14 09:43:02.670605478 +0000 UTC m=+0.434561457,LastTimestamp:2026-09-14 09:43:02.670605478 +0000 UTC m=+0.434561457,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-40.ec2.internal,}" Sep 14 09:43:02.686559 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.686544 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jxnxm" Sep 14 09:43:02.689446 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.689171 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-40.ec2.internal.18d526b446a1dba5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-40.ec2.internal,UID:ip-10-0-135-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-40.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-40.ec2.internal,},FirstTimestamp:2026-09-14 09:43:02.670621605 +0000 UTC m=+0.434577585,LastTimestamp:2026-09-14 09:43:02.670621605 +0000 UTC m=+0.434577585,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-40.ec2.internal,}" Sep 14 09:43:02.707429 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.707414 2550 manager.go:337] "Starting Device Plugin manager" Sep 14 09:43:02.720097 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.707448 2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 14 09:43:02.720097 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.707458 2550 server.go:87] "Starting device plugin registration server" Sep 14 09:43:02.720097 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.707660 2550 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 14 09:43:02.720097 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.707670 2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 14 09:43:02.720097 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.707850 2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 14 09:43:02.720097 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.707931 2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 14 09:43:02.720097 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.707940 2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 14 09:43:02.720097 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.708555 2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 14 09:43:02.720097 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.708593 2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:02.747596 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.747568 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 14 09:43:02.748713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.748689 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 14 09:43:02.748790 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.748743 2550 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 14 09:43:02.748790 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.748765 2550 kubelet.go:2448] "Starting kubelet main sync loop" Sep 14 09:43:02.748914 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.748835 2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 14 09:43:02.751374 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.751358 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 09:43:02.808523 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.808506 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 09:43:02.809259 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.809243 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 14 09:43:02.809331 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.809275 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 09:43:02.809331 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.809288 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 14 09:43:02.809331 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.809315 2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.819951 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.819916 2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.819951 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.819935 2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-40.ec2.internal\": node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:02.849230 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.849082 2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal"] Sep 14 09:43:02.849300 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.849262 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 09:43:02.850456 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.850444 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 14 09:43:02.850503 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.850470 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 09:43:02.850503 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.850480 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 14 09:43:02.851867 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.851855 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 09:43:02.851973 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.851960 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.852034 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.851984 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 09:43:02.852548 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.852533 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 14 09:43:02.852593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.852561 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 09:43:02.852593 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.852572 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 14 09:43:02.852652 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.852533 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 14 09:43:02.852652 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.852636 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 09:43:02.852652 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.852647 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 14 09:43:02.853862 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.853849 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.853909 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.853871 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 09:43:02.854464 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.854443 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31d7650ef8fb26af4fcd551e30be1116-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"31d7650ef8fb26af4fcd551e30be1116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.854540 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.854482 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31d7650ef8fb26af4fcd551e30be1116-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"31d7650ef8fb26af4fcd551e30be1116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.854540 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.854491 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 14 09:43:02.854540 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.854505 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/98f5720cef50a512c0a0bcbc18cd38fe-config\") pod \"kube-apiserver-proxy-ip-10-0-135-40.ec2.internal\" (UID: \"98f5720cef50a512c0a0bcbc18cd38fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.854540 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.854512 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 09:43:02.854540 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.854526 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 14 09:43:02.869012 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.868990 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:02.880325 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.880304 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-40.ec2.internal\" not found" node="ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.884470 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.884455 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-40.ec2.internal\" not found" node="ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.954738 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.954716 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31d7650ef8fb26af4fcd551e30be1116-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"31d7650ef8fb26af4fcd551e30be1116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.954738 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.954739 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/98f5720cef50a512c0a0bcbc18cd38fe-config\") pod \"kube-apiserver-proxy-ip-10-0-135-40.ec2.internal\" (UID: \"98f5720cef50a512c0a0bcbc18cd38fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.954892 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.954755 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31d7650ef8fb26af4fcd551e30be1116-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"31d7650ef8fb26af4fcd551e30be1116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.954892 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.954782 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31d7650ef8fb26af4fcd551e30be1116-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"31d7650ef8fb26af4fcd551e30be1116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.954892 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.954817 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31d7650ef8fb26af4fcd551e30be1116-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"31d7650ef8fb26af4fcd551e30be1116\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.954892 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:02.954825 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/98f5720cef50a512c0a0bcbc18cd38fe-config\") pod \"kube-apiserver-proxy-ip-10-0-135-40.ec2.internal\" (UID: \"98f5720cef50a512c0a0bcbc18cd38fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 14 09:43:02.969871 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:02.969852 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:03.070506 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:03.070446 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:03.171162 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:03.171134 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:03.183290 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:03.183274 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 14 09:43:03.188331 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:03.188317 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 14 09:43:04.340705 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:04.340681 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:04.441208 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:04.441155 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:04.541676 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:04.541655 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:04.589008 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.588991 2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 09:43:04.642517 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:04.642498 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:04.653665 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.653646 2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 14 09:43:04.653784 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.653765 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 09:43:04.653823 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.653797 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 09:43:04.743389 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:04.743332 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:04.762275 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.762241 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 09:38:03 +0000 UTC" deadline="2028-07-01 09:58:57.15419672 +0000 UTC" Sep 14 09:43:04.762275 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.762270 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15744h15m52.391929886s" Sep 14 09:43:04.844085 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:04.844057 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:04.855907 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:04.855875 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f5720cef50a512c0a0bcbc18cd38fe.slice/crio-1e8e85601f411ae085eaa126e4cfb3183d2b0cd6dde7e205a2d56bb3d07c0336 WatchSource:0}: Error finding container 1e8e85601f411ae085eaa126e4cfb3183d2b0cd6dde7e205a2d56bb3d07c0336: Status 404 returned error can't find the container with id 1e8e85601f411ae085eaa126e4cfb3183d2b0cd6dde7e205a2d56bb3d07c0336 Sep 14 09:43:04.860156 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:04.860133 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d7650ef8fb26af4fcd551e30be1116.slice/crio-dda6df55c953efa40697d76b13fe11e718da8bce5da2dfd21d5a96dca5a78b80 WatchSource:0}: Error finding container dda6df55c953efa40697d76b13fe11e718da8bce5da2dfd21d5a96dca5a78b80: Status 404 returned error can't find the container with id dda6df55c953efa40697d76b13fe11e718da8bce5da2dfd21d5a96dca5a78b80 Sep 14 09:43:04.862492 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.862473 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 09:43:04.913400 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.913379 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 09:43:04.914377 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.914363 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 14 09:43:04.914448 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.914390 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 09:43:04.914448 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.914407 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 14 09:43:04.945132 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:04.945110 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 14 09:43:04.996359 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:04.996303 2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 09:43:05.022464 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.022447 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 14 09:43:05.046148 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.046127 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 09:43:05.047180 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.047165 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 14 09:43:05.059880 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.059866 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 09:43:05.089657 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.089627 2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 09:43:05.699789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.699545 2550 apiserver.go:52] "Watching apiserver" Sep 14 09:43:05.709136 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.709107 2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 14 09:43:05.710223 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.710185 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-4htsc","openshift-image-registry/node-ca-phdx8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal","openshift-network-diagnostics/network-check-target-7hkpf","openshift-network-operator/iptables-alerter-cz5db","kube-system/global-pull-secret-syncer-z8prz","openshift-cluster-node-tuning-operator/tuned-288wc","openshift-multus/multus-additional-cni-plugins-59vql","openshift-multus/multus-w5pkh","openshift-multus/network-metrics-daemon-lnvrp","openshift-ovn-kubernetes/ovnkube-node-bzcvk","kube-system/konnectivity-agent-gfw87","kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn"] Sep 14 09:43:05.712038 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.712016 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.713574 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.713554 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.714571 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.714550 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.716255 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.716236 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:05.716332 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.716306 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:05.716789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.716770 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4whkh\"" Sep 14 09:43:05.717281 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.717023 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.717464 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.717525 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.717464 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.717982 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.718842 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-585bf\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.719068 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.719493 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.719521 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.719614 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.719769 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.719525 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9jhfc\"" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.719978 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.720036 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:05.721186 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.720434 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:05.721877 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.721507 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 14 09:43:05.722378 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.722034 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 14 09:43:05.722378 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.722075 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.723320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.723140 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:05.723320 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.723197 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:05.725659 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.725138 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.725659 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.725496 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 14 09:43:05.726144 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.725923 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8zf54\"" Sep 14 09:43:05.726144 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.726084 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 14 09:43:05.727686 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.727625 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:05.729718 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.728539 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 14 09:43:05.729718 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.728620 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 14 09:43:05.729718 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.728786 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bndtx\"" Sep 14 09:43:05.729718 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.728798 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nfpxs\"" Sep 14 09:43:05.729718 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.728929 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 14 09:43:05.729718 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.729132 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.731722 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed-hosts-file\") pod \"node-resolver-4htsc\" (UID: \"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed\") " pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.731769 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-daemon-config\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.731807 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.731838 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-sysctl-conf\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.731879 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-run\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.731936 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-device-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.731974 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-cni-dir\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732010 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-cnibin\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732051 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-run-multus-certs\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732103 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-cni-binary-copy\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732137 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48d8a147-3628-4352-8812-e5cd05abe9a5-host-slash\") pod \"iptables-alerter-cz5db\" (UID: \"48d8a147-3628-4352-8812-e5cd05abe9a5\") " pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732181 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-system-cni-dir\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732211 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-sysconfig\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732233 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-host\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732262 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39d283b1-6049-4e1d-957f-72c2674d9a8e-tmp\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.734320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732291 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48d8a147-3628-4352-8812-e5cd05abe9a5-iptables-alerter-script\") pod \"iptables-alerter-cz5db\" (UID: \"48d8a147-3628-4352-8812-e5cd05abe9a5\") " pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732328 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-socket-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732354 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq7c6\" (UniqueName: \"kubernetes.io/projected/0bcd4184-2a88-4b07-ace5-024964ef7925-kube-api-access-zq7c6\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732388 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-sysctl-d\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732440 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-run-netns\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732468 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-var-lib-kubelet\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732499 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxzth\" (UniqueName: \"kubernetes.io/projected/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-kube-api-access-jxzth\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732533 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0bcd4184-2a88-4b07-ace5-024964ef7925-cni-binary-copy\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732563 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhm8j\" (UniqueName: \"kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j\") pod \"network-check-target-7hkpf\" (UID: \"411da759-8982-4e40-9c15-66ba899c49e1\") " pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732567 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732594 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94f8bdb5-679a-4df1-9422-497f18ff1bbc-kubelet-config\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732620 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-registration-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732662 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed-tmp-dir\") pod \"node-resolver-4htsc\" (UID: \"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed\") " pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732701 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732692 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-var-lib-kubelet\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732770 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-system-cni-dir\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732799 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-sys\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.735130 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732886 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.732984 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-systemd\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733017 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7vfh\" (UniqueName: \"kubernetes.io/projected/48d8a147-3628-4352-8812-e5cd05abe9a5-kube-api-access-v7vfh\") pod \"iptables-alerter-cz5db\" (UID: \"48d8a147-3628-4352-8812-e5cd05abe9a5\") " pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733049 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733082 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-sys-fs\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733138 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-run-k8s-cni-cncf-io\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733170 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-var-lib-cni-bin\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733195 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733200 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-etc-kubernetes\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733231 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-modprobe-d\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733261 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhrv\" (UniqueName: \"kubernetes.io/projected/39d283b1-6049-4e1d-957f-72c2674d9a8e-kube-api-access-wkhrv\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733304 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-kubernetes\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733367 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqlh\" (UniqueName: \"kubernetes.io/projected/eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed-kube-api-access-2cqlh\") pod \"node-resolver-4htsc\" (UID: \"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed\") " pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733529 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-hostroot\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733571 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733619 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-socket-dir-parent\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733780 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.735884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733816 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94f8bdb5-679a-4df1-9422-497f18ff1bbc-dbus\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733918 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-conf-dir\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733946 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-tuned\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.733973 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grlq\" (UniqueName: \"kubernetes.io/projected/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-kube-api-access-2grlq\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734053 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734076 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5grc\" (UniqueName: \"kubernetes.io/projected/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-kube-api-access-m5grc\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734103 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734285 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mgvd9\"" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734321 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734352 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734105 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734496 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-var-lib-cni-multus\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734603 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lrgk4\"" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734613 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-cnibin\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734658 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-lib-modules\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.734661 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.736442 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.736520 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wh4fc\"" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.736616 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 14 09:43:05.736713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.736629 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 14 09:43:05.737800 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.736818 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 14 09:43:05.743736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.743709 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 14 09:43:05.795494 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.795470 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 09:43:05.799910 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.799888 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7j8kl" Sep 14 09:43:05.818253 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.817609 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7j8kl" Sep 14 09:43:05.822375 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.822330 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" event={"ID":"98f5720cef50a512c0a0bcbc18cd38fe","Type":"ContainerStarted","Data":"1e8e85601f411ae085eaa126e4cfb3183d2b0cd6dde7e205a2d56bb3d07c0336"} Sep 14 09:43:05.823382 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.823364 2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 14 09:43:05.823992 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.823970 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" event={"ID":"31d7650ef8fb26af4fcd551e30be1116","Type":"ContainerStarted","Data":"dda6df55c953efa40697d76b13fe11e718da8bce5da2dfd21d5a96dca5a78b80"} Sep 14 09:43:05.835435 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835413 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-run-k8s-cni-cncf-io\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.835533 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835447 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-var-lib-cni-bin\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.835533 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835471 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-etc-kubernetes\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.835533 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835496 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-modprobe-d\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.835533 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835520 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhrv\" (UniqueName: \"kubernetes.io/projected/39d283b1-6049-4e1d-957f-72c2674d9a8e-kube-api-access-wkhrv\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835529 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-run-k8s-cni-cncf-io\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835542 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-etc-kubernetes\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835547 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-var-lib-openvswitch\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835534 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-var-lib-cni-bin\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835575 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-cni-bin\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835600 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-kubernetes\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835625 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60182aa7-5db0-4cb3-b543-7b29dc4fafe3-konnectivity-ca\") pod \"konnectivity-agent-gfw87\" (UID: \"60182aa7-5db0-4cb3-b543-7b29dc4fafe3\") " pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835660 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-modprobe-d\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835664 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxgn4\" (UniqueName: \"kubernetes.io/projected/f6a256d1-4e71-4521-8cd9-2225f59b514e-kube-api-access-wxgn4\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835666 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-kubernetes\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.835780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835714 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqlh\" (UniqueName: \"kubernetes.io/projected/eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed-kube-api-access-2cqlh\") pod \"node-resolver-4htsc\" (UID: \"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed\") " pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835791 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-hostroot\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835819 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835847 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835859 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-hostroot\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835875 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-socket-dir-parent\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835905 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835930 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94f8bdb5-679a-4df1-9422-497f18ff1bbc-dbus\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835957 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60182aa7-5db0-4cb3-b543-7b29dc4fafe3-agent-certs\") pod \"konnectivity-agent-gfw87\" (UID: \"60182aa7-5db0-4cb3-b543-7b29dc4fafe3\") " pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.835981 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52a488f7-cd8d-4974-b6cb-5e293a21a2a6-host\") pod \"node-ca-phdx8\" (UID: \"52a488f7-cd8d-4974-b6cb-5e293a21a2a6\") " pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836009 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836005 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-run-systemd\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836038 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-socket-dir-parent\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836060 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-conf-dir\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836087 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-tuned\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836099 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94f8bdb5-679a-4df1-9422-497f18ff1bbc-dbus\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836119 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grlq\" (UniqueName: \"kubernetes.io/projected/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-kube-api-access-2grlq\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:05.836324 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836138 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-conf-dir\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836148 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836177 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-kubelet\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836209 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5grc\" (UniqueName: \"kubernetes.io/projected/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-kube-api-access-m5grc\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836226 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836238 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836394 2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836587 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-node-log\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836607 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836626 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-var-lib-cni-multus\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836674 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-cnibin\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836705 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-lib-modules\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836739 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4q48\" (UniqueName: \"kubernetes.io/projected/52a488f7-cd8d-4974-b6cb-5e293a21a2a6-kube-api-access-b4q48\") pod \"node-ca-phdx8\" (UID: \"52a488f7-cd8d-4974-b6cb-5e293a21a2a6\") " pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836822 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-cnibin\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836826 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836835 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-var-lib-cni-multus\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836862 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed-hosts-file\") pod \"node-resolver-4htsc\" (UID: \"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed\") " pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:05.837115 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836887 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-lib-modules\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836911 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed-hosts-file\") pod \"node-resolver-4htsc\" (UID: \"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed\") " pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836891 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-daemon-config\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836970 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.836996 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-sysctl-conf\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837022 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-run\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837068 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-run\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837186 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-sysctl-conf\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837225 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-device-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837252 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-run-openvswitch\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837284 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-cni-dir\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837307 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-cnibin\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837333 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-run-multus-certs\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837358 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-cni-binary-copy\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837383 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48d8a147-3628-4352-8812-e5cd05abe9a5-host-slash\") pod \"iptables-alerter-cz5db\" (UID: \"48d8a147-3628-4352-8812-e5cd05abe9a5\") " pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837379 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-device-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837410 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-systemd-units\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.837991 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837412 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-daemon-config\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837451 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-run-multus-certs\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837462 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-multus-cni-dir\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837472 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837503 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-cnibin\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837503 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-etc-openvswitch\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837543 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48d8a147-3628-4352-8812-e5cd05abe9a5-host-slash\") pod \"iptables-alerter-cz5db\" (UID: \"48d8a147-3628-4352-8812-e5cd05abe9a5\") " pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837539 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-cni-netd\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837578 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-system-cni-dir\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837600 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-sysconfig\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837620 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-host\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837688 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-sysconfig\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837695 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-host\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837725 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39d283b1-6049-4e1d-957f-72c2674d9a8e-tmp\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837757 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48d8a147-3628-4352-8812-e5cd05abe9a5-iptables-alerter-script\") pod \"iptables-alerter-cz5db\" (UID: \"48d8a147-3628-4352-8812-e5cd05abe9a5\") " pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837703 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-system-cni-dir\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837781 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-socket-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837818 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52a488f7-cd8d-4974-b6cb-5e293a21a2a6-serviceca\") pod \"node-ca-phdx8\" (UID: \"52a488f7-cd8d-4974-b6cb-5e293a21a2a6\") " pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:05.838755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837855 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7c6\" (UniqueName: \"kubernetes.io/projected/0bcd4184-2a88-4b07-ace5-024964ef7925-kube-api-access-zq7c6\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837879 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-sysctl-d\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837904 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-cni-binary-copy\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837903 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-run-ovn\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.837879 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-socket-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838075 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6a256d1-4e71-4521-8cd9-2225f59b514e-ovn-node-metrics-cert\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838108 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-sysctl-d\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838122 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-run-netns\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838150 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-var-lib-kubelet\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838168 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxzth\" (UniqueName: \"kubernetes.io/projected/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-kube-api-access-jxzth\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838183 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6a256d1-4e71-4521-8cd9-2225f59b514e-ovnkube-config\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838197 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-run-netns\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838242 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6a256d1-4e71-4521-8cd9-2225f59b514e-env-overrides\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838248 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-var-lib-kubelet\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838275 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0bcd4184-2a88-4b07-ace5-024964ef7925-cni-binary-copy\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838311 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm8j\" (UniqueName: \"kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j\") pod \"network-check-target-7hkpf\" (UID: \"411da759-8982-4e40-9c15-66ba899c49e1\") " pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838366 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94f8bdb5-679a-4df1-9422-497f18ff1bbc-kubelet-config\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:05.839670 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838407 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-registration-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838428 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48d8a147-3628-4352-8812-e5cd05abe9a5-iptables-alerter-script\") pod \"iptables-alerter-cz5db\" (UID: \"48d8a147-3628-4352-8812-e5cd05abe9a5\") " pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838431 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-slash\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838473 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94f8bdb5-679a-4df1-9422-497f18ff1bbc-kubelet-config\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838521 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed-tmp-dir\") pod \"node-resolver-4htsc\" (UID: \"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed\") " pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838544 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-var-lib-kubelet\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838576 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-registration-dir\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838622 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-system-cni-dir\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838671 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-system-cni-dir\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838748 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-sys\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838780 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838788 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0bcd4184-2a88-4b07-ace5-024964ef7925-cni-binary-copy\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838804 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bcd4184-2a88-4b07-ace5-024964ef7925-host-var-lib-kubelet\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838842 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-run-netns\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838857 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-sys\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.838949 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.838981 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-log-socket\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.840518 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.839022 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret podName:94f8bdb5-679a-4df1-9422-497f18ff1bbc nodeName:}" failed. No retries permitted until 2026-09-14 09:43:06.338995664 +0000 UTC m=+3.034120970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret") pod "global-pull-secret-syncer-z8prz" (UID: "94f8bdb5-679a-4df1-9422-497f18ff1bbc") : object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.839012 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.839052 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-systemd\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.839074 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7vfh\" (UniqueName: \"kubernetes.io/projected/48d8a147-3628-4352-8812-e5cd05abe9a5-kube-api-access-v7vfh\") pod \"iptables-alerter-cz5db\" (UID: \"48d8a147-3628-4352-8812-e5cd05abe9a5\") " pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.839093 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.839112 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6a256d1-4e71-4521-8cd9-2225f59b514e-ovnkube-script-lib\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.839115 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed-tmp-dir\") pod \"node-resolver-4htsc\" (UID: \"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed\") " pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.839134 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-sys-fs\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.839279 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-sys-fs\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.839289 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.839331 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-systemd\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.839408 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs podName:f4b05f60-7a5d-498c-9e7b-e0bc98d0d495 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:06.33939268 +0000 UTC m=+3.034517966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs") pod "network-metrics-daemon-lnvrp" (UID: "f4b05f60-7a5d-498c-9e7b-e0bc98d0d495") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.840626 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39d283b1-6049-4e1d-957f-72c2674d9a8e-tmp\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.841271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.840798 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39d283b1-6049-4e1d-957f-72c2674d9a8e-etc-tuned\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.849731 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.849660 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhrv\" (UniqueName: \"kubernetes.io/projected/39d283b1-6049-4e1d-957f-72c2674d9a8e-kube-api-access-wkhrv\") pod \"tuned-288wc\" (UID: \"39d283b1-6049-4e1d-957f-72c2674d9a8e\") " pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:05.856465 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.856442 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5grc\" (UniqueName: \"kubernetes.io/projected/b4a830d4-0f56-4eb4-a2d7-e967a5a22256-kube-api-access-m5grc\") pod \"multus-additional-cni-plugins-59vql\" (UID: \"b4a830d4-0f56-4eb4-a2d7-e967a5a22256\") " pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:05.858205 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.858183 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 09:43:05.858298 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.858212 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 09:43:05.858298 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.858234 2550 projected.go:196] Error preparing data for projected volume kube-api-access-xhm8j for pod openshift-network-diagnostics/network-check-target-7hkpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:05.858407 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:05.858298 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j podName:411da759-8982-4e40-9c15-66ba899c49e1 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:06.358271169 +0000 UTC m=+3.053396456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xhm8j" (UniqueName: "kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j") pod "network-check-target-7hkpf" (UID: "411da759-8982-4e40-9c15-66ba899c49e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:05.860338 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.860285 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqlh\" (UniqueName: \"kubernetes.io/projected/eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed-kube-api-access-2cqlh\") pod \"node-resolver-4htsc\" (UID: \"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed\") " pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:05.861628 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.861585 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grlq\" (UniqueName: \"kubernetes.io/projected/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-kube-api-access-2grlq\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:05.862001 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.861977 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxzth\" (UniqueName: \"kubernetes.io/projected/5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe-kube-api-access-jxzth\") pod \"aws-ebs-csi-driver-node-z64hn\" (UID: \"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:05.863358 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.863334 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq7c6\" (UniqueName: \"kubernetes.io/projected/0bcd4184-2a88-4b07-ace5-024964ef7925-kube-api-access-zq7c6\") pod \"multus-w5pkh\" (UID: \"0bcd4184-2a88-4b07-ace5-024964ef7925\") " pod="openshift-multus/multus-w5pkh" Sep 14 09:43:05.864063 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.864043 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7vfh\" (UniqueName: \"kubernetes.io/projected/48d8a147-3628-4352-8812-e5cd05abe9a5-kube-api-access-v7vfh\") pod \"iptables-alerter-cz5db\" (UID: \"48d8a147-3628-4352-8812-e5cd05abe9a5\") " pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:05.939782 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939748 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-node-log\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.939961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939795 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4q48\" (UniqueName: \"kubernetes.io/projected/52a488f7-cd8d-4974-b6cb-5e293a21a2a6-kube-api-access-b4q48\") pod \"node-ca-phdx8\" (UID: \"52a488f7-cd8d-4974-b6cb-5e293a21a2a6\") " pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:05.939961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939826 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-run-openvswitch\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.939961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939853 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-systemd-units\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.939961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939875 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-etc-openvswitch\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.939961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939875 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-node-log\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.939961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939897 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-cni-netd\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.939961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939940 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-cni-netd\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.939961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939943 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52a488f7-cd8d-4974-b6cb-5e293a21a2a6-serviceca\") pod \"node-ca-phdx8\" (UID: \"52a488f7-cd8d-4974-b6cb-5e293a21a2a6\") " pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939970 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-run-ovn\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.939996 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6a256d1-4e71-4521-8cd9-2225f59b514e-ovn-node-metrics-cert\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940013 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-run-openvswitch\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940024 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6a256d1-4e71-4521-8cd9-2225f59b514e-ovnkube-config\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940049 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6a256d1-4e71-4521-8cd9-2225f59b514e-env-overrides\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940079 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-etc-openvswitch\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940090 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-slash\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940128 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-run-netns\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940153 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-log-socket\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940175 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940242 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6a256d1-4e71-4521-8cd9-2225f59b514e-ovnkube-script-lib\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940273 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-var-lib-openvswitch\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940297 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-cni-bin\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.940339 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940323 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60182aa7-5db0-4cb3-b543-7b29dc4fafe3-konnectivity-ca\") pod \"konnectivity-agent-gfw87\" (UID: \"60182aa7-5db0-4cb3-b543-7b29dc4fafe3\") " pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:05.941020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940351 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxgn4\" (UniqueName: \"kubernetes.io/projected/f6a256d1-4e71-4521-8cd9-2225f59b514e-kube-api-access-wxgn4\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940380 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940409 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60182aa7-5db0-4cb3-b543-7b29dc4fafe3-agent-certs\") pod \"konnectivity-agent-gfw87\" (UID: \"60182aa7-5db0-4cb3-b543-7b29dc4fafe3\") " pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:05.941020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940433 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52a488f7-cd8d-4974-b6cb-5e293a21a2a6-host\") pod \"node-ca-phdx8\" (UID: \"52a488f7-cd8d-4974-b6cb-5e293a21a2a6\") " pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:05.941020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940442 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-run-ovn\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940457 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-run-systemd\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940490 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-kubelet\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940562 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-kubelet\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940947 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52a488f7-cd8d-4974-b6cb-5e293a21a2a6-serviceca\") pod \"node-ca-phdx8\" (UID: \"52a488f7-cd8d-4974-b6cb-5e293a21a2a6\") " pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:05.941020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941009 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-var-lib-openvswitch\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941052 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-run-netns\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941055 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-log-socket\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941085 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.940050 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-systemd-units\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941123 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6a256d1-4e71-4521-8cd9-2225f59b514e-ovnkube-config\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941138 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-slash\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941181 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-cni-bin\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941580 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6a256d1-4e71-4521-8cd9-2225f59b514e-ovnkube-script-lib\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941715 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60182aa7-5db0-4cb3-b543-7b29dc4fafe3-konnectivity-ca\") pod \"konnectivity-agent-gfw87\" (UID: \"60182aa7-5db0-4cb3-b543-7b29dc4fafe3\") " pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:05.941815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941777 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-run-systemd\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941776 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52a488f7-cd8d-4974-b6cb-5e293a21a2a6-host\") pod \"node-ca-phdx8\" (UID: \"52a488f7-cd8d-4974-b6cb-5e293a21a2a6\") " pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:05.941996 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941885 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6a256d1-4e71-4521-8cd9-2225f59b514e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.941996 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.941934 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6a256d1-4e71-4521-8cd9-2225f59b514e-env-overrides\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.942949 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.942920 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6a256d1-4e71-4521-8cd9-2225f59b514e-ovn-node-metrics-cert\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:05.944321 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:05.944298 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60182aa7-5db0-4cb3-b543-7b29dc4fafe3-agent-certs\") pod \"konnectivity-agent-gfw87\" (UID: \"60182aa7-5db0-4cb3-b543-7b29dc4fafe3\") " pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:06.005422 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.005353 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxgn4\" (UniqueName: \"kubernetes.io/projected/f6a256d1-4e71-4521-8cd9-2225f59b514e-kube-api-access-wxgn4\") pod \"ovnkube-node-bzcvk\" (UID: \"f6a256d1-4e71-4521-8cd9-2225f59b514e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:06.005422 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.005366 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4q48\" (UniqueName: \"kubernetes.io/projected/52a488f7-cd8d-4974-b6cb-5e293a21a2a6-kube-api-access-b4q48\") pod \"node-ca-phdx8\" (UID: \"52a488f7-cd8d-4974-b6cb-5e293a21a2a6\") " pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:06.022087 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.022065 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 09:43:06.033356 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.033333 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-59vql" Sep 14 09:43:06.044819 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:06.044785 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a830d4_0f56_4eb4_a2d7_e967a5a22256.slice/crio-a604fa13814ca8608054dd0026950cdb2b327c007684b714be54da62769da1a4 WatchSource:0}: Error finding container a604fa13814ca8608054dd0026950cdb2b327c007684b714be54da62769da1a4: Status 404 returned error can't find the container with id a604fa13814ca8608054dd0026950cdb2b327c007684b714be54da62769da1a4 Sep 14 09:43:06.048566 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.048546 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" Sep 14 09:43:06.052507 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.052484 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-288wc" Sep 14 09:43:06.057878 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:06.057854 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5154f2e5_3dbb_4d3b_8353_49c6f62ebdfe.slice/crio-e5508dbc16be8ed7157bd2a0e5b267a59b8efa22db1d04e610f159cf09dbe993 WatchSource:0}: Error finding container e5508dbc16be8ed7157bd2a0e5b267a59b8efa22db1d04e610f159cf09dbe993: Status 404 returned error can't find the container with id e5508dbc16be8ed7157bd2a0e5b267a59b8efa22db1d04e610f159cf09dbe993 Sep 14 09:43:06.060836 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:06.060812 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d283b1_6049_4e1d_957f_72c2674d9a8e.slice/crio-e463c65d88e11913cf8180c748da73fe0762931e8bf00f96de9491e12b97adfb WatchSource:0}: Error finding container e463c65d88e11913cf8180c748da73fe0762931e8bf00f96de9491e12b97adfb: Status 404 returned error can't find the container with id e463c65d88e11913cf8180c748da73fe0762931e8bf00f96de9491e12b97adfb Sep 14 09:43:06.061900 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.061880 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cz5db" Sep 14 09:43:06.071812 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:06.071788 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d8a147_3628_4352_8812_e5cd05abe9a5.slice/crio-ef720e199fd2f490fc8b72cb6109afa1e3ff5dd3280df7ec35f47ebaab95eb1f WatchSource:0}: Error finding container ef720e199fd2f490fc8b72cb6109afa1e3ff5dd3280df7ec35f47ebaab95eb1f: Status 404 returned error can't find the container with id ef720e199fd2f490fc8b72cb6109afa1e3ff5dd3280df7ec35f47ebaab95eb1f Sep 14 09:43:06.075619 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.075249 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4htsc" Sep 14 09:43:06.082951 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.082933 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w5pkh" Sep 14 09:43:06.091030 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.091007 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:06.098800 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.098780 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:06.106294 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.106275 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-phdx8" Sep 14 09:43:06.342507 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.342415 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:06.342507 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.342463 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:06.342730 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:06.342577 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:06.342730 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:06.342660 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret podName:94f8bdb5-679a-4df1-9422-497f18ff1bbc nodeName:}" failed. No retries permitted until 2026-09-14 09:43:07.342626818 +0000 UTC m=+4.037752105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret") pod "global-pull-secret-syncer-z8prz" (UID: "94f8bdb5-679a-4df1-9422-497f18ff1bbc") : object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:06.342730 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:06.342579 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:06.342891 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:06.342733 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs podName:f4b05f60-7a5d-498c-9e7b-e0bc98d0d495 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:07.342717117 +0000 UTC m=+4.037842417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs") pod "network-metrics-daemon-lnvrp" (UID: "f4b05f60-7a5d-498c-9e7b-e0bc98d0d495") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:06.443039 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.443009 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm8j\" (UniqueName: \"kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j\") pod \"network-check-target-7hkpf\" (UID: \"411da759-8982-4e40-9c15-66ba899c49e1\") " pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:06.443207 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:06.443148 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 09:43:06.443207 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:06.443163 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 09:43:06.443207 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:06.443173 2550 projected.go:196] Error preparing data for projected volume kube-api-access-xhm8j for pod openshift-network-diagnostics/network-check-target-7hkpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:06.443349 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:06.443221 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j podName:411da759-8982-4e40-9c15-66ba899c49e1 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:07.443208335 +0000 UTC m=+4.138333617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhm8j" (UniqueName: "kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j") pod "network-check-target-7hkpf" (UID: "411da759-8982-4e40-9c15-66ba899c49e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:06.818773 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.818700 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:06.819463 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:06.818832 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:06.819666 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.819617 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 09:38:05 +0000 UTC" deadline="2028-06-17 06:41:05.466096462 +0000 UTC" Sep 14 09:43:06.819666 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.819663 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15404h57m58.646438873s" Sep 14 09:43:06.827618 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.827295 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cz5db" event={"ID":"48d8a147-3628-4352-8812-e5cd05abe9a5","Type":"ContainerStarted","Data":"ef720e199fd2f490fc8b72cb6109afa1e3ff5dd3280df7ec35f47ebaab95eb1f"} Sep 14 09:43:06.829049 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.829023 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-288wc" event={"ID":"39d283b1-6049-4e1d-957f-72c2674d9a8e","Type":"ContainerStarted","Data":"e463c65d88e11913cf8180c748da73fe0762931e8bf00f96de9491e12b97adfb"} Sep 14 09:43:06.830379 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.830352 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" event={"ID":"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe","Type":"ContainerStarted","Data":"e5508dbc16be8ed7157bd2a0e5b267a59b8efa22db1d04e610f159cf09dbe993"} Sep 14 09:43:06.831649 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:06.831610 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59vql" event={"ID":"b4a830d4-0f56-4eb4-a2d7-e967a5a22256","Type":"ContainerStarted","Data":"a604fa13814ca8608054dd0026950cdb2b327c007684b714be54da62769da1a4"} Sep 14 09:43:07.101420 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.101335 2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 09:43:07.129649 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:07.129578 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60182aa7_5db0_4cb3_b543_7b29dc4fafe3.slice/crio-54da1d9a71babd76c6d8b71d27d60586b6748ba9fb4ae77deea3d2b3db900b6d WatchSource:0}: Error finding container 54da1d9a71babd76c6d8b71d27d60586b6748ba9fb4ae77deea3d2b3db900b6d: Status 404 returned error can't find the container with id 54da1d9a71babd76c6d8b71d27d60586b6748ba9fb4ae77deea3d2b3db900b6d Sep 14 09:43:07.132333 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:07.132221 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a488f7_cd8d_4974_b6cb_5e293a21a2a6.slice/crio-af79e718791825705e1e389b40b94a1d807f1b84aecfe41fbc2f5721f190aefd WatchSource:0}: Error finding container af79e718791825705e1e389b40b94a1d807f1b84aecfe41fbc2f5721f190aefd: Status 404 returned error can't find the container with id af79e718791825705e1e389b40b94a1d807f1b84aecfe41fbc2f5721f190aefd Sep 14 09:43:07.133936 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:07.133100 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a256d1_4e71_4521_8cd9_2225f59b514e.slice/crio-990bcac8e1e0f5672f6f90322529103ca414571d161a099b1595306ffea33c77 WatchSource:0}: Error finding container 990bcac8e1e0f5672f6f90322529103ca414571d161a099b1595306ffea33c77: Status 404 returned error can't find the container with id 990bcac8e1e0f5672f6f90322529103ca414571d161a099b1595306ffea33c77 Sep 14 09:43:07.134149 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:07.134132 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcd4184_2a88_4b07_ace5_024964ef7925.slice/crio-f4e5edd3519da7a8a7c0b7f98d9e7d1bf7b6906d82417334afa0bf01b927b9e8 WatchSource:0}: Error finding container f4e5edd3519da7a8a7c0b7f98d9e7d1bf7b6906d82417334afa0bf01b927b9e8: Status 404 returned error can't find the container with id f4e5edd3519da7a8a7c0b7f98d9e7d1bf7b6906d82417334afa0bf01b927b9e8 Sep 14 09:43:07.135183 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:07.135161 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae3b65a_f2f9_49e5_83ce_6bb3e6cb57ed.slice/crio-7e6f088d3c700363e51e1a9a2c7ceb0360e80ab490dac8d81a5c557a1c2f91b3 WatchSource:0}: Error finding container 7e6f088d3c700363e51e1a9a2c7ceb0360e80ab490dac8d81a5c557a1c2f91b3: Status 404 returned error can't find the container with id 7e6f088d3c700363e51e1a9a2c7ceb0360e80ab490dac8d81a5c557a1c2f91b3 Sep 14 09:43:07.350755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.350501 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:07.350924 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:07.350778 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:07.350924 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.350787 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:07.350924 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:07.350856 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret podName:94f8bdb5-679a-4df1-9422-497f18ff1bbc nodeName:}" failed. No retries permitted until 2026-09-14 09:43:09.350832349 +0000 UTC m=+6.045957635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret") pod "global-pull-secret-syncer-z8prz" (UID: "94f8bdb5-679a-4df1-9422-497f18ff1bbc") : object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:07.350924 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:07.350902 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:07.351141 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:07.350954 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs podName:f4b05f60-7a5d-498c-9e7b-e0bc98d0d495 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:09.350937942 +0000 UTC m=+6.046063227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs") pod "network-metrics-daemon-lnvrp" (UID: "f4b05f60-7a5d-498c-9e7b-e0bc98d0d495") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:07.452037 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.451957 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm8j\" (UniqueName: \"kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j\") pod \"network-check-target-7hkpf\" (UID: \"411da759-8982-4e40-9c15-66ba899c49e1\") " pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:07.452192 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:07.452151 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 09:43:07.452192 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:07.452172 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 09:43:07.452192 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:07.452184 2550 projected.go:196] Error preparing data for projected volume kube-api-access-xhm8j for pod openshift-network-diagnostics/network-check-target-7hkpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:07.452352 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:07.452237 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j podName:411da759-8982-4e40-9c15-66ba899c49e1 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:09.452220217 +0000 UTC m=+6.147345504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhm8j" (UniqueName: "kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j") pod "network-check-target-7hkpf" (UID: "411da759-8982-4e40-9c15-66ba899c49e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:07.819047 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.818667 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:07.819047 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:07.818816 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:07.819629 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.819182 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:07.819629 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:07.819291 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:07.820308 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.820261 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 09:38:05 +0000 UTC" deadline="2028-02-29 07:28:45.705922112 +0000 UTC" Sep 14 09:43:07.820414 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.820308 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12789h45m37.885617405s" Sep 14 09:43:07.855602 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.855569 2550 generic.go:356] "Generic (PLEG): container finished" podID="31d7650ef8fb26af4fcd551e30be1116" containerID="15fdb45358e13429f7cc183263f7a0bb04e166a411ffd0a510b9ab836f536f9d" exitCode=0 Sep 14 09:43:07.855761 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.855676 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" event={"ID":"31d7650ef8fb26af4fcd551e30be1116","Type":"ContainerDied","Data":"15fdb45358e13429f7cc183263f7a0bb04e166a411ffd0a510b9ab836f536f9d"} Sep 14 09:43:07.878335 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.878294 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" event={"ID":"98f5720cef50a512c0a0bcbc18cd38fe","Type":"ContainerStarted","Data":"7984e2d4fcb37dc274e11cc41d80329ffd1f47e72f92c8b5cd1f9fa99ec9c115"} Sep 14 09:43:07.883224 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.883197 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4htsc" event={"ID":"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed","Type":"ContainerStarted","Data":"7e6f088d3c700363e51e1a9a2c7ceb0360e80ab490dac8d81a5c557a1c2f91b3"} Sep 14 09:43:07.894676 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.894625 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w5pkh" event={"ID":"0bcd4184-2a88-4b07-ace5-024964ef7925","Type":"ContainerStarted","Data":"f4e5edd3519da7a8a7c0b7f98d9e7d1bf7b6906d82417334afa0bf01b927b9e8"} Sep 14 09:43:07.902729 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.902703 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" event={"ID":"f6a256d1-4e71-4521-8cd9-2225f59b514e","Type":"ContainerStarted","Data":"990bcac8e1e0f5672f6f90322529103ca414571d161a099b1595306ffea33c77"} Sep 14 09:43:07.913226 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.913201 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-phdx8" event={"ID":"52a488f7-cd8d-4974-b6cb-5e293a21a2a6","Type":"ContainerStarted","Data":"af79e718791825705e1e389b40b94a1d807f1b84aecfe41fbc2f5721f190aefd"} Sep 14 09:43:07.915508 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:07.915484 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gfw87" event={"ID":"60182aa7-5db0-4cb3-b543-7b29dc4fafe3","Type":"ContainerStarted","Data":"54da1d9a71babd76c6d8b71d27d60586b6748ba9fb4ae77deea3d2b3db900b6d"} Sep 14 09:43:08.818553 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:08.818477 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:08.818732 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:08.818607 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:09.369711 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:09.369663 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:09.370159 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:09.369719 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:09.370159 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:09.369863 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:09.370159 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:09.369923 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs podName:f4b05f60-7a5d-498c-9e7b-e0bc98d0d495 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:13.36990494 +0000 UTC m=+10.065030229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs") pod "network-metrics-daemon-lnvrp" (UID: "f4b05f60-7a5d-498c-9e7b-e0bc98d0d495") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:09.370342 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:09.370327 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:09.370402 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:09.370378 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret podName:94f8bdb5-679a-4df1-9422-497f18ff1bbc nodeName:}" failed. No retries permitted until 2026-09-14 09:43:13.370363004 +0000 UTC m=+10.065488290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret") pod "global-pull-secret-syncer-z8prz" (UID: "94f8bdb5-679a-4df1-9422-497f18ff1bbc") : object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:09.470340 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:09.470301 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm8j\" (UniqueName: \"kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j\") pod \"network-check-target-7hkpf\" (UID: \"411da759-8982-4e40-9c15-66ba899c49e1\") " pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:09.470526 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:09.470509 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 09:43:09.470591 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:09.470534 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 09:43:09.470591 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:09.470546 2550 projected.go:196] Error preparing data for projected volume kube-api-access-xhm8j for pod openshift-network-diagnostics/network-check-target-7hkpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:09.470730 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:09.470605 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j podName:411da759-8982-4e40-9c15-66ba899c49e1 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:13.470586836 +0000 UTC m=+10.165712122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhm8j" (UniqueName: "kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j") pod "network-check-target-7hkpf" (UID: "411da759-8982-4e40-9c15-66ba899c49e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:09.818734 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:09.818204 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:09.818734 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:09.818333 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:09.818734 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:09.818398 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:09.818734 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:09.818515 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:10.818478 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:10.818444 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:10.818962 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:10.818915 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:11.819136 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:11.818846 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:11.819136 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:11.818974 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:11.819136 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:11.818814 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:11.819786 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:11.819180 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:12.819284 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:12.818761 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:12.819284 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:12.818885 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:13.407273 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:13.406516 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:13.407273 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:13.406566 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:13.407273 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:13.406748 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:13.407273 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:13.406815 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs podName:f4b05f60-7a5d-498c-9e7b-e0bc98d0d495 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:21.406794076 +0000 UTC m=+18.101919361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs") pod "network-metrics-daemon-lnvrp" (UID: "f4b05f60-7a5d-498c-9e7b-e0bc98d0d495") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:13.407273 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:13.407187 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:13.407273 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:13.407242 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret podName:94f8bdb5-679a-4df1-9422-497f18ff1bbc nodeName:}" failed. No retries permitted until 2026-09-14 09:43:21.407228849 +0000 UTC m=+18.102354135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret") pod "global-pull-secret-syncer-z8prz" (UID: "94f8bdb5-679a-4df1-9422-497f18ff1bbc") : object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:13.507933 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:13.507892 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm8j\" (UniqueName: \"kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j\") pod \"network-check-target-7hkpf\" (UID: \"411da759-8982-4e40-9c15-66ba899c49e1\") " pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:13.508120 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:13.508017 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 09:43:13.508120 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:13.508038 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 09:43:13.508120 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:13.508053 2550 projected.go:196] Error preparing data for projected volume kube-api-access-xhm8j for pod openshift-network-diagnostics/network-check-target-7hkpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:13.508120 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:13.508114 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j podName:411da759-8982-4e40-9c15-66ba899c49e1 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:21.508096946 +0000 UTC m=+18.203222252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhm8j" (UniqueName: "kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j") pod "network-check-target-7hkpf" (UID: "411da759-8982-4e40-9c15-66ba899c49e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:13.819738 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:13.818966 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:13.819738 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:13.819073 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:13.819738 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:13.819471 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:13.819738 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:13.819648 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:14.817930 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:14.817847 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:14.818151 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:14.817952 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:15.818015 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:15.817977 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:15.818429 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:15.818029 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:15.818429 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:15.818142 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:15.818429 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:15.818260 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:16.818371 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:16.818340 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:16.818699 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:16.818443 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:17.818445 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:17.818415 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:17.818883 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:17.818533 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:17.818883 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:17.818573 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:17.818883 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:17.818661 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:18.818759 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:18.818719 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:18.819143 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:18.818848 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:19.818057 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:19.818026 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:19.818218 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:19.818025 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:19.818218 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:19.818165 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:19.818321 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:19.818221 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:20.817799 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:20.817725 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:20.818205 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:20.817825 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:21.465662 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:21.465612 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:21.465844 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:21.465674 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:21.465844 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:21.465766 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:21.465844 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:21.465783 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:21.465844 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:21.465833 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret podName:94f8bdb5-679a-4df1-9422-497f18ff1bbc nodeName:}" failed. No retries permitted until 2026-09-14 09:43:37.465814787 +0000 UTC m=+34.160940089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret") pod "global-pull-secret-syncer-z8prz" (UID: "94f8bdb5-679a-4df1-9422-497f18ff1bbc") : object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:21.466021 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:21.465853 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs podName:f4b05f60-7a5d-498c-9e7b-e0bc98d0d495 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:37.465842996 +0000 UTC m=+34.160968285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs") pod "network-metrics-daemon-lnvrp" (UID: "f4b05f60-7a5d-498c-9e7b-e0bc98d0d495") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:21.566683 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:21.566627 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 09:43:21.566683 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:21.566676 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 09:43:21.566683 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:21.566675 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm8j\" (UniqueName: \"kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j\") pod \"network-check-target-7hkpf\" (UID: \"411da759-8982-4e40-9c15-66ba899c49e1\") " pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:21.566946 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:21.566693 2550 projected.go:196] Error preparing data for projected volume kube-api-access-xhm8j for pod openshift-network-diagnostics/network-check-target-7hkpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:21.566946 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:21.566838 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j podName:411da759-8982-4e40-9c15-66ba899c49e1 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:37.566819272 +0000 UTC m=+34.261944559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhm8j" (UniqueName: "kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j") pod "network-check-target-7hkpf" (UID: "411da759-8982-4e40-9c15-66ba899c49e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:21.818299 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:21.818210 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:21.818743 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:21.818329 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:21.818743 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:21.818381 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:21.818743 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:21.818494 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:22.818399 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:22.818365 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:22.818891 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:22.818495 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:23.819436 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:23.819405 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:23.819886 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:23.819515 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:23.819886 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:23.819560 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:23.819886 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:23.819691 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:24.818831 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.818671 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:24.818926 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:24.818902 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:24.954649 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.954477 2550 generic.go:356] "Generic (PLEG): container finished" podID="b4a830d4-0f56-4eb4-a2d7-e967a5a22256" containerID="3372c33c418e1578aec925ac378fae0535a82f1e1e7916f99b0e01de83e47769" exitCode=0 Sep 14 09:43:24.955211 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.954566 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59vql" event={"ID":"b4a830d4-0f56-4eb4-a2d7-e967a5a22256","Type":"ContainerDied","Data":"3372c33c418e1578aec925ac378fae0535a82f1e1e7916f99b0e01de83e47769"} Sep 14 09:43:24.956596 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.956567 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4htsc" event={"ID":"eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed","Type":"ContainerStarted","Data":"af9a358553f6438d670d80e0dea6819fcc351a8567f4b41452797dfc73562043"} Sep 14 09:43:24.958610 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.958586 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w5pkh" event={"ID":"0bcd4184-2a88-4b07-ace5-024964ef7925","Type":"ContainerStarted","Data":"733751749cf4a328d7d22fbfd90861b6828837d12e7196d542357b53926fe56a"} Sep 14 09:43:24.960870 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.960843 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-288wc" event={"ID":"39d283b1-6049-4e1d-957f-72c2674d9a8e","Type":"ContainerStarted","Data":"29a54540f8b09c33c967070f315df848fca76773c956955605a88823c149c26d"} Sep 14 09:43:24.962971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.962951 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" event={"ID":"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe","Type":"ContainerStarted","Data":"02f93f078163159a9088ac9646e19e1bbc150fee21476bc6d587b7c4f560a8f7"} Sep 14 09:43:24.964912 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.964889 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" event={"ID":"f6a256d1-4e71-4521-8cd9-2225f59b514e","Type":"ContainerStarted","Data":"d1a57bb69037f1c8dddb255f865cb42db649dd2b773b9f89f509bc40c3debec2"} Sep 14 09:43:24.964997 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.964947 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" event={"ID":"f6a256d1-4e71-4521-8cd9-2225f59b514e","Type":"ContainerStarted","Data":"9680e8f5ff8e77fae3ccc341313fe30de198f258661bb4f963240be57ac8c216"} Sep 14 09:43:24.966528 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.966472 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-phdx8" event={"ID":"52a488f7-cd8d-4974-b6cb-5e293a21a2a6","Type":"ContainerStarted","Data":"674afe9d6bffa74ff89a08f5e193bcee7c95aee260514a1d3ea7f18c33a825c6"} Sep 14 09:43:24.968089 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.968066 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gfw87" event={"ID":"60182aa7-5db0-4cb3-b543-7b29dc4fafe3","Type":"ContainerStarted","Data":"7ca56745ac18fd423bc23f8d3aca3ee4ce107d10cbdd92ddc4d92651ab280edb"} Sep 14 09:43:24.969988 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.969970 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" event={"ID":"31d7650ef8fb26af4fcd551e30be1116","Type":"ContainerStarted","Data":"6de3c4113d5613d5d4fcca9bb9d4afe28c3e30867082e39132ac271f0ecb3ea1"} Sep 14 09:43:24.981422 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:24.981374 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" podStartSLOduration=19.981362324 podStartE2EDuration="19.981362324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:07.911903568 +0000 UTC m=+4.607028874" watchObservedRunningTime="2026-09-14 09:43:24.981362324 +0000 UTC m=+21.676487629" Sep 14 09:43:25.004873 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.004834 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gfw87" podStartSLOduration=3.884150331 podStartE2EDuration="21.00481948s" totalImagesPullingTime="17.120669149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:25.004197432 +0000 UTC m=+21.699322738" watchObservedRunningTime="2026-09-14 09:43:25.00481948 +0000 UTC m=+21.699944784" Sep 14 09:43:25.032566 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.032521 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-phdx8" podStartSLOduration=3.930798056 podStartE2EDuration="21.032502385s" totalImagesPullingTime="17.101704329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:25.031328591 +0000 UTC m=+21.726453897" watchObservedRunningTime="2026-09-14 09:43:25.032502385 +0000 UTC m=+21.727627690" Sep 14 09:43:25.071461 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.071419 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-288wc" podStartSLOduration=2.882565457 podStartE2EDuration="21.071406385s" totalImagesPullingTime="18.188840928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:25.071030445 +0000 UTC m=+21.766155751" watchObservedRunningTime="2026-09-14 09:43:25.071406385 +0000 UTC m=+21.766531691" Sep 14 09:43:25.099532 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.099481 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4htsc" podStartSLOduration=4.001403049 podStartE2EDuration="21.09946188s" totalImagesPullingTime="17.098058831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:25.098554931 +0000 UTC m=+21.793680236" watchObservedRunningTime="2026-09-14 09:43:25.09946188 +0000 UTC m=+21.794587186" Sep 14 09:43:25.125986 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.125942 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w5pkh" podStartSLOduration=3.9631011210000002 podStartE2EDuration="21.125925734s" totalImagesPullingTime="17.162824613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:25.125916341 +0000 UTC m=+21.821041646" watchObservedRunningTime="2026-09-14 09:43:25.125925734 +0000 UTC m=+21.821051038" Sep 14 09:43:25.151160 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.151104 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" podStartSLOduration=20.151082387 podStartE2EDuration="20.151082387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:25.150874481 +0000 UTC m=+21.845999787" watchObservedRunningTime="2026-09-14 09:43:25.151082387 +0000 UTC m=+21.846207694" Sep 14 09:43:25.401431 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.401408 2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 14 09:43:25.788966 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.788775 2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T09:43:25.401428131Z","UUID":"47bc1d5c-f772-410e-9ad2-783d1a610d0f","Handler":null,"Name":"","Endpoint":""} Sep 14 09:43:25.791366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.791347 2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 14 09:43:25.791489 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.791377 2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 14 09:43:25.818319 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.818289 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:25.818457 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:25.818418 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:25.818533 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.818471 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:25.818615 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:25.818590 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:25.975482 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.974828 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cz5db" event={"ID":"48d8a147-3628-4352-8812-e5cd05abe9a5","Type":"ContainerStarted","Data":"ffb0c65f604970a82d64649f6fdc871160f61a4da77b271517468619b03b9102"} Sep 14 09:43:25.977960 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.977931 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" event={"ID":"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe","Type":"ContainerStarted","Data":"6fd38f0b7f40cb7cc2161a7b18cc4ce533e0cf59a10ead499d1c9190cc035ef3"} Sep 14 09:43:25.982386 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.982359 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" event={"ID":"f6a256d1-4e71-4521-8cd9-2225f59b514e","Type":"ContainerStarted","Data":"3f3e263abc85f9c2405f5fd9f55b700627a3dc83c84f1c510e82f8c447b53768"} Sep 14 09:43:25.982533 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.982436 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" event={"ID":"f6a256d1-4e71-4521-8cd9-2225f59b514e","Type":"ContainerStarted","Data":"c6ccf2f6e298fb106fe8966ac021fdf3d0728efb1162dd12e260302e835a28fd"} Sep 14 09:43:25.982533 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.982485 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" event={"ID":"f6a256d1-4e71-4521-8cd9-2225f59b514e","Type":"ContainerStarted","Data":"386e877bf94f95304d227c54408526cf93fb2f25615f5443306a67e77b2d3ed9"} Sep 14 09:43:25.982533 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:25.982531 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" event={"ID":"f6a256d1-4e71-4521-8cd9-2225f59b514e","Type":"ContainerStarted","Data":"3547dccdd6806fac253b357652fb488ae9cfb61603f120f2847637da5f552205"} Sep 14 09:43:26.019057 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:26.019001 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cz5db" podStartSLOduration=3.856672722 podStartE2EDuration="22.018982253s" totalImagesPullingTime="18.162309531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:26.018769572 +0000 UTC m=+22.713894878" watchObservedRunningTime="2026-09-14 09:43:26.018982253 +0000 UTC m=+22.714107560" Sep 14 09:43:26.818269 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:26.818198 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:26.818404 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:26.818295 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:26.990059 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:26.990016 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" event={"ID":"5154f2e5-3dbb-4d3b-8353-49c6f62ebdfe","Type":"ContainerStarted","Data":"02ee8ec749a049f2f06d7e445c9784cd964eafbf501a817cb4206828fc6880da"} Sep 14 09:43:27.015701 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:27.015649 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z64hn" podStartSLOduration=4.075156083 podStartE2EDuration="24.015616525s" totalImagesPullingTime="19.940460442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:27.014994579 +0000 UTC m=+23.710119931" watchObservedRunningTime="2026-09-14 09:43:27.015616525 +0000 UTC m=+23.710741831" Sep 14 09:43:27.819044 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:27.818854 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:27.819308 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:27.818854 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:27.819308 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:27.819160 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:27.819308 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:27.819187 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:27.996306 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:27.996274 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" event={"ID":"f6a256d1-4e71-4521-8cd9-2225f59b514e","Type":"ContainerStarted","Data":"8a33660aa3c68beeec2a3389da2222b170c1273636b9428a89685b38bbdcc86f"} Sep 14 09:43:28.818576 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:28.818541 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:28.818766 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:28.818690 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:29.095936 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:29.095863 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:29.096517 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:29.096497 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:29.818473 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:29.818299 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:29.818605 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:29.818347 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:29.818605 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:29.818574 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:29.818712 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:29.818616 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:30.002304 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:30.002273 2550 generic.go:356] "Generic (PLEG): container finished" podID="b4a830d4-0f56-4eb4-a2d7-e967a5a22256" containerID="fb01d0965eea3402bc793c1dd21a68f4666c8cfa700ac1699957e4635646ac0d" exitCode=0 Sep 14 09:43:30.002446 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:30.002357 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59vql" event={"ID":"b4a830d4-0f56-4eb4-a2d7-e967a5a22256","Type":"ContainerDied","Data":"fb01d0965eea3402bc793c1dd21a68f4666c8cfa700ac1699957e4635646ac0d"} Sep 14 09:43:30.006468 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:30.006440 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" event={"ID":"f6a256d1-4e71-4521-8cd9-2225f59b514e","Type":"ContainerStarted","Data":"ad1943cacdea60bf202ec3a5d0ad845b2ab6447e2683324111ea575691577a9e"} Sep 14 09:43:30.006705 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:30.006678 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:30.007151 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:30.007130 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gfw87" Sep 14 09:43:30.817867 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:30.817843 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:30.818407 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:30.817970 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:31.010961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.010887 2550 generic.go:356] "Generic (PLEG): container finished" podID="b4a830d4-0f56-4eb4-a2d7-e967a5a22256" containerID="1d15e0e3ead62d467d0413673f828f1fff24d32331e5602558e8e75fd9c60815" exitCode=0 Sep 14 09:43:31.011105 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.010954 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59vql" event={"ID":"b4a830d4-0f56-4eb4-a2d7-e967a5a22256","Type":"ContainerDied","Data":"1d15e0e3ead62d467d0413673f828f1fff24d32331e5602558e8e75fd9c60815"} Sep 14 09:43:31.011598 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.011562 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:31.011737 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.011644 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:31.011737 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.011689 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:31.025459 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.025440 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:31.025564 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.025554 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:43:31.048923 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.048881 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" podStartSLOduration=9.374567107 podStartE2EDuration="27.048867338s" totalImagesPullingTime="17.674300231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:30.095041005 +0000 UTC m=+26.790166309" watchObservedRunningTime="2026-09-14 09:43:31.048867338 +0000 UTC m=+27.743992642" Sep 14 09:43:31.526354 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.526318 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7hkpf"] Sep 14 09:43:31.526518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.526440 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:31.526583 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:31.526534 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:31.530135 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.530097 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z8prz"] Sep 14 09:43:31.530274 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.530182 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:31.530332 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:31.530278 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:31.530669 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.530630 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lnvrp"] Sep 14 09:43:31.530787 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:31.530742 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:31.530860 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:31.530839 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:32.818408 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:32.818240 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:32.818784 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:32.818468 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:33.018112 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:33.018081 2550 generic.go:356] "Generic (PLEG): container finished" podID="b4a830d4-0f56-4eb4-a2d7-e967a5a22256" containerID="e60f8ca6290a7c49d64d44d507c026dc4f20d5a2c245d4223676dab0b9dfcfe7" exitCode=0 Sep 14 09:43:33.018236 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:33.018148 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59vql" event={"ID":"b4a830d4-0f56-4eb4-a2d7-e967a5a22256","Type":"ContainerDied","Data":"e60f8ca6290a7c49d64d44d507c026dc4f20d5a2c245d4223676dab0b9dfcfe7"} Sep 14 09:43:33.818731 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:33.818704 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:33.819161 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:33.818812 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:33.819161 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:33.818856 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:33.819161 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:33.818984 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:34.818431 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:34.818403 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:34.818623 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:34.818505 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:35.818620 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:35.818590 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:35.819353 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:35.818747 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnvrp" podUID="f4b05f60-7a5d-498c-9e7b-e0bc98d0d495" Sep 14 09:43:35.819353 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:35.818799 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:35.819353 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:35.818928 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8prz" podUID="94f8bdb5-679a-4df1-9422-497f18ff1bbc" Sep 14 09:43:36.818060 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:36.818025 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:36.818294 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:36.818158 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hkpf" podUID="411da759-8982-4e40-9c15-66ba899c49e1" Sep 14 09:43:37.059114 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.059084 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeReady" Sep 14 09:43:37.059750 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.059240 2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 14 09:43:37.103650 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.103560 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6956cc7fb6-pmxc7"] Sep 14 09:43:37.123424 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.123393 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.128272 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.128246 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 14 09:43:37.128272 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.128262 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 14 09:43:37.128481 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.128344 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 14 09:43:37.128563 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.128547 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xchsg\"" Sep 14 09:43:37.135438 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.135418 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 14 09:43:37.135901 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.135884 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6956cc7fb6-pmxc7"] Sep 14 09:43:37.141084 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.140979 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-85622"] Sep 14 09:43:37.161761 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.161739 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q5n48"] Sep 14 09:43:37.161890 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.161856 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-85622" Sep 14 09:43:37.164479 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.164454 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 14 09:43:37.164652 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.164617 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 14 09:43:37.164806 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.164787 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wswml\"" Sep 14 09:43:37.164915 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.164893 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 14 09:43:37.181965 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.181946 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q5n48"] Sep 14 09:43:37.182061 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.181974 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-85622"] Sep 14 09:43:37.182061 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.182039 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.188268 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.188248 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 14 09:43:37.188787 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.188774 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 14 09:43:37.189418 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.189403 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwxbr\"" Sep 14 09:43:37.282727 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.282704 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-config-volume\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.282836 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.282739 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-tmp-dir\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.282836 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.282759 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b94aee3-35c3-45ee-aa73-dd0083f07b78-ca-trust-extracted\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.282836 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.282775 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-certificates\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.282836 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.282791 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-bound-sa-token\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.282836 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.282809 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-image-registry-private-configuration\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.282836 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.282825 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.283094 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.282839 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-installation-pull-secrets\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.283094 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.282947 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-trusted-ca\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.283094 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.282980 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9b6r\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-kube-api-access-n9b6r\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.283094 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.283012 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.283094 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.283057 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gww7\" (UniqueName: \"kubernetes.io/projected/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-kube-api-access-7gww7\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.283094 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.283081 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbgz\" (UniqueName: \"kubernetes.io/projected/53089d2e-4275-4a32-96e6-5800b3f71b08-kube-api-access-hkbgz\") pod \"ingress-canary-85622\" (UID: \"53089d2e-4275-4a32-96e6-5800b3f71b08\") " pod="openshift-ingress-canary/ingress-canary-85622" Sep 14 09:43:37.283279 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.283128 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53089d2e-4275-4a32-96e6-5800b3f71b08-cert\") pod \"ingress-canary-85622\" (UID: \"53089d2e-4275-4a32-96e6-5800b3f71b08\") " pod="openshift-ingress-canary/ingress-canary-85622" Sep 14 09:43:37.384006 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.383984 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.384095 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384020 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gww7\" (UniqueName: \"kubernetes.io/projected/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-kube-api-access-7gww7\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.384095 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384041 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbgz\" (UniqueName: \"kubernetes.io/projected/53089d2e-4275-4a32-96e6-5800b3f71b08-kube-api-access-hkbgz\") pod \"ingress-canary-85622\" (UID: \"53089d2e-4275-4a32-96e6-5800b3f71b08\") " pod="openshift-ingress-canary/ingress-canary-85622" Sep 14 09:43:37.384177 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384090 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53089d2e-4275-4a32-96e6-5800b3f71b08-cert\") pod \"ingress-canary-85622\" (UID: \"53089d2e-4275-4a32-96e6-5800b3f71b08\") " pod="openshift-ingress-canary/ingress-canary-85622" Sep 14 09:43:37.384177 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384127 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-config-volume\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.384262 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384174 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-tmp-dir\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.384262 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384200 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b94aee3-35c3-45ee-aa73-dd0083f07b78-ca-trust-extracted\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.384262 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384223 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-certificates\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.384496 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.384302 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 09:43:37.384496 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.384361 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls podName:a7e4ee1c-60de-43ff-9b68-507c7ff58bdb nodeName:}" failed. No retries permitted until 2026-09-14 09:43:37.884342875 +0000 UTC m=+34.579468171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls") pod "dns-default-q5n48" (UID: "a7e4ee1c-60de-43ff-9b68-507c7ff58bdb") : secret "dns-default-metrics-tls" not found Sep 14 09:43:37.384603 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384537 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-bound-sa-token\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.384603 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384557 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-tmp-dir\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.384603 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384578 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-image-registry-private-configuration\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.384772 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384605 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.384772 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384629 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-installation-pull-secrets\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.384772 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384627 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b94aee3-35c3-45ee-aa73-dd0083f07b78-ca-trust-extracted\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.384772 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384732 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-trusted-ca\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.384772 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384761 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9b6r\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-kube-api-access-n9b6r\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.385015 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.384854 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 09:43:37.385015 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.384868 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6956cc7fb6-pmxc7: secret "image-registry-tls" not found Sep 14 09:43:37.385015 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.384911 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls podName:8b94aee3-35c3-45ee-aa73-dd0083f07b78 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:37.884898478 +0000 UTC m=+34.580023761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls") pod "image-registry-6956cc7fb6-pmxc7" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78") : secret "image-registry-tls" not found Sep 14 09:43:37.385015 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384924 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-certificates\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.385015 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.384956 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-config-volume\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.385735 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.385715 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-trusted-ca\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.388224 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.388201 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-installation-pull-secrets\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.388310 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.388267 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53089d2e-4275-4a32-96e6-5800b3f71b08-cert\") pod \"ingress-canary-85622\" (UID: \"53089d2e-4275-4a32-96e6-5800b3f71b08\") " pod="openshift-ingress-canary/ingress-canary-85622" Sep 14 09:43:37.388310 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.388265 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-image-registry-private-configuration\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.396711 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.396689 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9b6r\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-kube-api-access-n9b6r\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.397798 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.397779 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-bound-sa-token\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.398120 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.398101 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbgz\" (UniqueName: \"kubernetes.io/projected/53089d2e-4275-4a32-96e6-5800b3f71b08-kube-api-access-hkbgz\") pod \"ingress-canary-85622\" (UID: \"53089d2e-4275-4a32-96e6-5800b3f71b08\") " pod="openshift-ingress-canary/ingress-canary-85622" Sep 14 09:43:37.402897 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.402876 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gww7\" (UniqueName: \"kubernetes.io/projected/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-kube-api-access-7gww7\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.471343 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.471321 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-85622" Sep 14 09:43:37.486010 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.485992 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:37.486098 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.486024 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:37.486159 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.486128 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:37.486159 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.486138 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:37.486249 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.486190 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs podName:f4b05f60-7a5d-498c-9e7b-e0bc98d0d495 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:09.486172058 +0000 UTC m=+66.181297345 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs") pod "network-metrics-daemon-lnvrp" (UID: "f4b05f60-7a5d-498c-9e7b-e0bc98d0d495") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 09:43:37.486249 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.486208 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret podName:94f8bdb5-679a-4df1-9422-497f18ff1bbc nodeName:}" failed. No retries permitted until 2026-09-14 09:44:09.486198697 +0000 UTC m=+66.181323985 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret") pod "global-pull-secret-syncer-z8prz" (UID: "94f8bdb5-679a-4df1-9422-497f18ff1bbc") : object "kube-system"/"original-pull-secret" not registered Sep 14 09:43:37.586775 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.586626 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm8j\" (UniqueName: \"kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j\") pod \"network-check-target-7hkpf\" (UID: \"411da759-8982-4e40-9c15-66ba899c49e1\") " pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:37.586922 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.586780 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 09:43:37.586922 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.586802 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 09:43:37.586922 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.586811 2550 projected.go:196] Error preparing data for projected volume kube-api-access-xhm8j for pod openshift-network-diagnostics/network-check-target-7hkpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:37.586922 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.586860 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j podName:411da759-8982-4e40-9c15-66ba899c49e1 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:09.586846778 +0000 UTC m=+66.281972065 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhm8j" (UniqueName: "kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j") pod "network-check-target-7hkpf" (UID: "411da759-8982-4e40-9c15-66ba899c49e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 09:43:37.779468 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.779441 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-85622"] Sep 14 09:43:37.782536 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:43:37.782503 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53089d2e_4275_4a32_96e6_5800b3f71b08.slice/crio-062c2545f5cdd140559d2e4212212c738bacc5c4ce407873d4d6250c1e2e0cdb WatchSource:0}: Error finding container 062c2545f5cdd140559d2e4212212c738bacc5c4ce407873d4d6250c1e2e0cdb: Status 404 returned error can't find the container with id 062c2545f5cdd140559d2e4212212c738bacc5c4ce407873d4d6250c1e2e0cdb Sep 14 09:43:37.818826 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.818806 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:43:37.818911 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.818804 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:43:37.828180 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.828160 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 09:43:37.828285 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.828187 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 09:43:37.828389 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.828377 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ctbn\"" Sep 14 09:43:37.889591 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.889538 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:37.889709 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:37.889597 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:37.889771 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.889713 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 09:43:37.889771 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.889729 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6956cc7fb6-pmxc7: secret "image-registry-tls" not found Sep 14 09:43:37.889771 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.889735 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 09:43:37.889888 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.889792 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls podName:8b94aee3-35c3-45ee-aa73-dd0083f07b78 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:38.88977326 +0000 UTC m=+35.584898544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls") pod "image-registry-6956cc7fb6-pmxc7" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78") : secret "image-registry-tls" not found Sep 14 09:43:37.889888 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:37.889814 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls podName:a7e4ee1c-60de-43ff-9b68-507c7ff58bdb nodeName:}" failed. No retries permitted until 2026-09-14 09:43:38.889804079 +0000 UTC m=+35.584929388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls") pod "dns-default-q5n48" (UID: "a7e4ee1c-60de-43ff-9b68-507c7ff58bdb") : secret "dns-default-metrics-tls" not found Sep 14 09:43:38.031307 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:38.031274 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-85622" event={"ID":"53089d2e-4275-4a32-96e6-5800b3f71b08","Type":"ContainerStarted","Data":"062c2545f5cdd140559d2e4212212c738bacc5c4ce407873d4d6250c1e2e0cdb"} Sep 14 09:43:38.036157 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:38.036133 2550 generic.go:356] "Generic (PLEG): container finished" podID="b4a830d4-0f56-4eb4-a2d7-e967a5a22256" containerID="323492eebf6019c0d18295926ead0767ec6d612cb1003dc3eed69ce3ab843db4" exitCode=0 Sep 14 09:43:38.036299 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:38.036170 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59vql" event={"ID":"b4a830d4-0f56-4eb4-a2d7-e967a5a22256","Type":"ContainerDied","Data":"323492eebf6019c0d18295926ead0767ec6d612cb1003dc3eed69ce3ab843db4"} Sep 14 09:43:38.818224 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:38.818193 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:43:38.823541 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:38.823515 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-clrv6\"" Sep 14 09:43:38.823708 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:38.823519 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 09:43:38.823708 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:38.823564 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 09:43:38.895860 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:38.895833 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:38.896001 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:38.895905 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:38.896001 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:38.895982 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 09:43:38.896163 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:38.896026 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 09:43:38.896163 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:38.896040 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6956cc7fb6-pmxc7: secret "image-registry-tls" not found Sep 14 09:43:38.896163 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:38.896047 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls podName:a7e4ee1c-60de-43ff-9b68-507c7ff58bdb nodeName:}" failed. No retries permitted until 2026-09-14 09:43:40.896030164 +0000 UTC m=+37.591155452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls") pod "dns-default-q5n48" (UID: "a7e4ee1c-60de-43ff-9b68-507c7ff58bdb") : secret "dns-default-metrics-tls" not found Sep 14 09:43:38.896163 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:38.896087 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls podName:8b94aee3-35c3-45ee-aa73-dd0083f07b78 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:40.896071088 +0000 UTC m=+37.591196393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls") pod "image-registry-6956cc7fb6-pmxc7" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78") : secret "image-registry-tls" not found Sep 14 09:43:39.043020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:39.042978 2550 generic.go:356] "Generic (PLEG): container finished" podID="b4a830d4-0f56-4eb4-a2d7-e967a5a22256" containerID="ec242e41b74e54682f86375decb9b099538b68edb37aad773644c87876fad6d5" exitCode=0 Sep 14 09:43:39.043192 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:39.043040 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59vql" event={"ID":"b4a830d4-0f56-4eb4-a2d7-e967a5a22256","Type":"ContainerDied","Data":"ec242e41b74e54682f86375decb9b099538b68edb37aad773644c87876fad6d5"} Sep 14 09:43:40.049849 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:40.049811 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59vql" event={"ID":"b4a830d4-0f56-4eb4-a2d7-e967a5a22256","Type":"ContainerStarted","Data":"d3566811ad7a4ab72ae78029c8b03688fd1dd4f292263a2b563f674bd00c5be2"} Sep 14 09:43:40.108034 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:40.107974 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-59vql" podStartSLOduration=12.157089556 podStartE2EDuration="37.107951325s" totalImagesPullingTime="24.950861769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:40.096093297 +0000 UTC m=+36.791218621" watchObservedRunningTime="2026-09-14 09:43:40.107951325 +0000 UTC m=+36.803076631" Sep 14 09:43:40.649002 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:40.648969 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfw87_60182aa7-5db0-4cb3-b543-7b29dc4fafe3/konnectivity-agent/0.log" limit=1024 Sep 14 09:43:40.908081 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:40.907898 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:40.908081 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:40.908068 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 09:43:40.908248 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:40.908082 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:40.908248 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:40.908134 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls podName:a7e4ee1c-60de-43ff-9b68-507c7ff58bdb nodeName:}" failed. No retries permitted until 2026-09-14 09:43:44.908112892 +0000 UTC m=+41.603238174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls") pod "dns-default-q5n48" (UID: "a7e4ee1c-60de-43ff-9b68-507c7ff58bdb") : secret "dns-default-metrics-tls" not found Sep 14 09:43:40.908248 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:40.908184 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 09:43:40.908248 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:40.908194 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6956cc7fb6-pmxc7: secret "image-registry-tls" not found Sep 14 09:43:40.908248 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:40.908232 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls podName:8b94aee3-35c3-45ee-aa73-dd0083f07b78 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:44.908222306 +0000 UTC m=+41.603347589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls") pod "image-registry-6956cc7fb6-pmxc7" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78") : secret "image-registry-tls" not found Sep 14 09:43:41.054069 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:41.054030 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-85622" event={"ID":"53089d2e-4275-4a32-96e6-5800b3f71b08","Type":"ContainerStarted","Data":"09484d41509fc1175d815399ad1ff7e0d027d0c4d3d3394a2252f84c2924adf0"} Sep 14 09:43:41.079821 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:41.079780 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-85622" podStartSLOduration=1.07769054 podStartE2EDuration="4.07976298s" totalImagesPullingTime="3.00207244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:41.079723445 +0000 UTC m=+37.774848752" watchObservedRunningTime="2026-09-14 09:43:41.07976298 +0000 UTC m=+37.774888284" Sep 14 09:43:42.268906 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:42.268875 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfw87_60182aa7-5db0-4cb3-b543-7b29dc4fafe3/konnectivity-agent/0.log" limit=1024 Sep 14 09:43:42.710203 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:42.710173 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfw87_60182aa7-5db0-4cb3-b543-7b29dc4fafe3/konnectivity-agent/0.log" limit=1024 Sep 14 09:43:42.857920 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:42.857892 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfw87_60182aa7-5db0-4cb3-b543-7b29dc4fafe3/konnectivity-agent/0.log" limit=1024 Sep 14 09:43:43.357649 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:43.357585 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfw87_60182aa7-5db0-4cb3-b543-7b29dc4fafe3/konnectivity-agent/0.log" limit=1024 Sep 14 09:43:43.433427 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:43.433399 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfw87_60182aa7-5db0-4cb3-b543-7b29dc4fafe3/konnectivity-agent/0.log" limit=1024 Sep 14 09:43:44.935364 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:44.935324 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:44.935808 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:44.935403 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:44.935808 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:44.935475 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 09:43:44.935808 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:44.935494 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6956cc7fb6-pmxc7: secret "image-registry-tls" not found Sep 14 09:43:44.935808 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:44.935530 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 09:43:44.935808 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:44.935558 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls podName:8b94aee3-35c3-45ee-aa73-dd0083f07b78 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:52.935541775 +0000 UTC m=+49.630667062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls") pod "image-registry-6956cc7fb6-pmxc7" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78") : secret "image-registry-tls" not found Sep 14 09:43:44.935808 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:44.935580 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls podName:a7e4ee1c-60de-43ff-9b68-507c7ff58bdb nodeName:}" failed. No retries permitted until 2026-09-14 09:43:52.935565108 +0000 UTC m=+49.630690404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls") pod "dns-default-q5n48" (UID: "a7e4ee1c-60de-43ff-9b68-507c7ff58bdb") : secret "dns-default-metrics-tls" not found Sep 14 09:43:46.501120 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:46.501088 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfw87_60182aa7-5db0-4cb3-b543-7b29dc4fafe3/konnectivity-agent/0.log" limit=1024 Sep 14 09:43:46.577943 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:46.577914 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfw87_60182aa7-5db0-4cb3-b543-7b29dc4fafe3/konnectivity-agent/0.log" limit=1024 Sep 14 09:43:52.991053 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:52.991022 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:43:52.991437 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:43:52.991074 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:43:52.991437 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:52.991164 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 09:43:52.991437 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:52.991177 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6956cc7fb6-pmxc7: secret "image-registry-tls" not found Sep 14 09:43:52.991437 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:52.991163 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 09:43:52.991437 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:52.991239 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls podName:8b94aee3-35c3-45ee-aa73-dd0083f07b78 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:08.991225355 +0000 UTC m=+65.686350638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls") pod "image-registry-6956cc7fb6-pmxc7" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78") : secret "image-registry-tls" not found Sep 14 09:43:52.991437 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:43:52.991304 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls podName:a7e4ee1c-60de-43ff-9b68-507c7ff58bdb nodeName:}" failed. No retries permitted until 2026-09-14 09:44:08.991288882 +0000 UTC m=+65.686414167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls") pod "dns-default-q5n48" (UID: "a7e4ee1c-60de-43ff-9b68-507c7ff58bdb") : secret "dns-default-metrics-tls" not found Sep 14 09:44:03.028274 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:03.028238 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzcvk" Sep 14 09:44:09.002037 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.001996 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:44:09.002524 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.002059 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:44:09.002524 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:09.002168 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 09:44:09.002524 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:09.002171 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 09:44:09.002524 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:09.002178 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6956cc7fb6-pmxc7: secret "image-registry-tls" not found Sep 14 09:44:09.002524 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:09.002268 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls podName:a7e4ee1c-60de-43ff-9b68-507c7ff58bdb nodeName:}" failed. No retries permitted until 2026-09-14 09:44:41.00224554 +0000 UTC m=+97.697370824 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls") pod "dns-default-q5n48" (UID: "a7e4ee1c-60de-43ff-9b68-507c7ff58bdb") : secret "dns-default-metrics-tls" not found Sep 14 09:44:09.002524 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:09.002323 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls podName:8b94aee3-35c3-45ee-aa73-dd0083f07b78 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:41.002312587 +0000 UTC m=+97.697437869 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls") pod "image-registry-6956cc7fb6-pmxc7" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78") : secret "image-registry-tls" not found Sep 14 09:44:09.505953 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.505908 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:44:09.505953 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.505956 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:44:09.513536 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.513517 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 09:44:09.514047 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.514035 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 09:44:09.517156 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:09.517140 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 09:44:09.517213 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:09.517203 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs podName:f4b05f60-7a5d-498c-9e7b-e0bc98d0d495 nodeName:}" failed. No retries permitted until 2026-09-14 09:45:13.517188625 +0000 UTC m=+130.212313908 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs") pod "network-metrics-daemon-lnvrp" (UID: "f4b05f60-7a5d-498c-9e7b-e0bc98d0d495") : secret "metrics-daemon-secret" not found Sep 14 09:44:09.520693 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.520672 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94f8bdb5-679a-4df1-9422-497f18ff1bbc-original-pull-secret\") pod \"global-pull-secret-syncer-z8prz\" (UID: \"94f8bdb5-679a-4df1-9422-497f18ff1bbc\") " pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:44:09.607100 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.607066 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm8j\" (UniqueName: \"kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j\") pod \"network-check-target-7hkpf\" (UID: \"411da759-8982-4e40-9c15-66ba899c49e1\") " pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:44:09.610098 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.610082 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 09:44:09.622519 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.622501 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 09:44:09.628116 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.628099 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8prz" Sep 14 09:44:09.630450 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.630419 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhm8j\" (UniqueName: \"kubernetes.io/projected/411da759-8982-4e40-9c15-66ba899c49e1-kube-api-access-xhm8j\") pod \"network-check-target-7hkpf\" (UID: \"411da759-8982-4e40-9c15-66ba899c49e1\") " pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:44:09.735526 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.735496 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-clrv6\"" Sep 14 09:44:09.742765 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.742749 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:44:09.902940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:09.902910 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z8prz"] Sep 14 09:44:09.910891 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:44:09.910868 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f8bdb5_679a_4df1_9422_497f18ff1bbc.slice/crio-5407f6bf51889ff189c8f0db36725af9683d0ed2a0fea970688cf5667ffeb413 WatchSource:0}: Error finding container 5407f6bf51889ff189c8f0db36725af9683d0ed2a0fea970688cf5667ffeb413: Status 404 returned error can't find the container with id 5407f6bf51889ff189c8f0db36725af9683d0ed2a0fea970688cf5667ffeb413 Sep 14 09:44:10.018373 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:10.018353 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7hkpf"] Sep 14 09:44:10.020384 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:44:10.020359 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411da759_8982_4e40_9c15_66ba899c49e1.slice/crio-21747575eb2a254f9a2bff4ef000418feecad5f5cd92c1f6696cbdf527ed14c7 WatchSource:0}: Error finding container 21747575eb2a254f9a2bff4ef000418feecad5f5cd92c1f6696cbdf527ed14c7: Status 404 returned error can't find the container with id 21747575eb2a254f9a2bff4ef000418feecad5f5cd92c1f6696cbdf527ed14c7 Sep 14 09:44:10.118922 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:10.118857 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7hkpf" event={"ID":"411da759-8982-4e40-9c15-66ba899c49e1","Type":"ContainerStarted","Data":"21747575eb2a254f9a2bff4ef000418feecad5f5cd92c1f6696cbdf527ed14c7"} Sep 14 09:44:10.119948 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:10.119928 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z8prz" event={"ID":"94f8bdb5-679a-4df1-9422-497f18ff1bbc","Type":"ContainerStarted","Data":"5407f6bf51889ff189c8f0db36725af9683d0ed2a0fea970688cf5667ffeb413"} Sep 14 09:44:15.134843 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:15.134807 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z8prz" event={"ID":"94f8bdb5-679a-4df1-9422-497f18ff1bbc","Type":"ContainerStarted","Data":"1fdc94d6668ff8f69ac4637a08a8504bc78bd8ca477e04a7b45391928e70dfba"} Sep 14 09:44:15.136237 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:15.136212 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7hkpf" event={"ID":"411da759-8982-4e40-9c15-66ba899c49e1","Type":"ContainerStarted","Data":"15e87d1af7b89153e328860fadc5adb457d8521978d2a22c40177a3d39bc7372"} Sep 14 09:44:15.136350 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:15.136334 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:44:15.154917 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:15.154872 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z8prz" podStartSLOduration=66.685908551 podStartE2EDuration="1m11.15485943s" totalImagesPullingTime="4.468950879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:15.154119562 +0000 UTC m=+71.849244868" watchObservedRunningTime="2026-09-14 09:44:15.15485943 +0000 UTC m=+71.849984734" Sep 14 09:44:15.181417 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:15.181379 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7hkpf" podStartSLOduration=66.825745826 podStartE2EDuration="1m11.181362664s" totalImagesPullingTime="4.355616838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:15.178181311 +0000 UTC m=+71.873306616" watchObservedRunningTime="2026-09-14 09:44:15.181362664 +0000 UTC m=+71.876487971" Sep 14 09:44:30.994077 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:30.994046 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht"] Sep 14 09:44:30.998242 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:30.998222 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.026880 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.026848 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 14 09:44:31.027035 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.026900 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 14 09:44:31.029082 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.029064 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 14 09:44:31.029276 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.029262 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w5hjm\"" Sep 14 09:44:31.029353 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.029304 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 14 09:44:31.042773 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.042750 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-76v9c"] Sep 14 09:44:31.045608 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.045593 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.047165 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.047143 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht"] Sep 14 09:44:31.061455 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.061439 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c6eb03b-bccc-48c8-81e5-94cae21943c6-serving-cert\") pod \"service-ca-operator-cf7c776b7-wm6ht\" (UID: \"5c6eb03b-bccc-48c8-81e5-94cae21943c6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.061538 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.061474 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24f7f\" (UniqueName: \"kubernetes.io/projected/5c6eb03b-bccc-48c8-81e5-94cae21943c6-kube-api-access-24f7f\") pod \"service-ca-operator-cf7c776b7-wm6ht\" (UID: \"5c6eb03b-bccc-48c8-81e5-94cae21943c6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.061538 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.061506 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c6eb03b-bccc-48c8-81e5-94cae21943c6-config\") pod \"service-ca-operator-cf7c776b7-wm6ht\" (UID: \"5c6eb03b-bccc-48c8-81e5-94cae21943c6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.067382 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.067368 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 14 09:44:31.069086 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.069068 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jgvfb\"" Sep 14 09:44:31.069127 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.069089 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 14 09:44:31.070880 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.070867 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 14 09:44:31.071017 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.070991 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 14 09:44:31.087945 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.087919 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 14 09:44:31.161934 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.161905 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a2690c3-8c2f-4124-8575-e75a9a44d107-config\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.161934 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.161933 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a2690c3-8c2f-4124-8575-e75a9a44d107-serving-cert\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.162099 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.161957 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c6eb03b-bccc-48c8-81e5-94cae21943c6-serving-cert\") pod \"service-ca-operator-cf7c776b7-wm6ht\" (UID: \"5c6eb03b-bccc-48c8-81e5-94cae21943c6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.162099 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.161986 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24f7f\" (UniqueName: \"kubernetes.io/projected/5c6eb03b-bccc-48c8-81e5-94cae21943c6-kube-api-access-24f7f\") pod \"service-ca-operator-cf7c776b7-wm6ht\" (UID: \"5c6eb03b-bccc-48c8-81e5-94cae21943c6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.162099 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.162013 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c6eb03b-bccc-48c8-81e5-94cae21943c6-config\") pod \"service-ca-operator-cf7c776b7-wm6ht\" (UID: \"5c6eb03b-bccc-48c8-81e5-94cae21943c6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.162099 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.162031 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a2690c3-8c2f-4124-8575-e75a9a44d107-trusted-ca\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.162099 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.162090 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlss\" (UniqueName: \"kubernetes.io/projected/6a2690c3-8c2f-4124-8575-e75a9a44d107-kube-api-access-jvlss\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.162569 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.162546 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c6eb03b-bccc-48c8-81e5-94cae21943c6-config\") pod \"service-ca-operator-cf7c776b7-wm6ht\" (UID: \"5c6eb03b-bccc-48c8-81e5-94cae21943c6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.164080 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.164055 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c6eb03b-bccc-48c8-81e5-94cae21943c6-serving-cert\") pod \"service-ca-operator-cf7c776b7-wm6ht\" (UID: \"5c6eb03b-bccc-48c8-81e5-94cae21943c6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.184769 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.184746 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-76v9c"] Sep 14 09:44:31.200242 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.200215 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24f7f\" (UniqueName: \"kubernetes.io/projected/5c6eb03b-bccc-48c8-81e5-94cae21943c6-kube-api-access-24f7f\") pod \"service-ca-operator-cf7c776b7-wm6ht\" (UID: \"5c6eb03b-bccc-48c8-81e5-94cae21943c6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.213764 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.213740 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7"] Sep 14 09:44:31.216669 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.216653 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7" Sep 14 09:44:31.220924 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:31.220902 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-40.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-cluster-storage-operator\": no relationship found between node 'ip-10-0-135-40.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 14 09:44:31.220989 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:31.220941 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"volume-data-source-validator-dockercfg-m5gcv\" is forbidden: User \"system:node:ip-10-0-135-40.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-cluster-storage-operator\": no relationship found between node 'ip-10-0-135-40.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-m5gcv\"" type="*v1.Secret" Sep 14 09:44:31.220989 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:31.220973 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-40.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-cluster-storage-operator\": no relationship found between node 'ip-10-0-135-40.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 14 09:44:31.248534 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.248477 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7"] Sep 14 09:44:31.262871 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.262847 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a2690c3-8c2f-4124-8575-e75a9a44d107-trusted-ca\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.262978 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.262897 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvzdh\" (UniqueName: \"kubernetes.io/projected/2d3ccc69-de69-4209-a388-ce7d1c1b45b9-kube-api-access-kvzdh\") pod \"volume-data-source-validator-86695b7cc7-n8hq7\" (UID: \"2d3ccc69-de69-4209-a388-ce7d1c1b45b9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7" Sep 14 09:44:31.262978 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.262918 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlss\" (UniqueName: \"kubernetes.io/projected/6a2690c3-8c2f-4124-8575-e75a9a44d107-kube-api-access-jvlss\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.262978 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.262938 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a2690c3-8c2f-4124-8575-e75a9a44d107-config\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.262978 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.262955 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a2690c3-8c2f-4124-8575-e75a9a44d107-serving-cert\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.263649 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.263613 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a2690c3-8c2f-4124-8575-e75a9a44d107-config\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.263863 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.263841 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a2690c3-8c2f-4124-8575-e75a9a44d107-trusted-ca\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.265164 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.265144 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a2690c3-8c2f-4124-8575-e75a9a44d107-serving-cert\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.306631 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.306610 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" Sep 14 09:44:31.359007 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.358982 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlss\" (UniqueName: \"kubernetes.io/projected/6a2690c3-8c2f-4124-8575-e75a9a44d107-kube-api-access-jvlss\") pod \"console-operator-696884fb9c-76v9c\" (UID: \"6a2690c3-8c2f-4124-8575-e75a9a44d107\") " pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.363604 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.363580 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvzdh\" (UniqueName: \"kubernetes.io/projected/2d3ccc69-de69-4209-a388-ce7d1c1b45b9-kube-api-access-kvzdh\") pod \"volume-data-source-validator-86695b7cc7-n8hq7\" (UID: \"2d3ccc69-de69-4209-a388-ce7d1c1b45b9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7" Sep 14 09:44:31.650523 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.650493 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht"] Sep 14 09:44:31.651398 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:44:31.651374 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6eb03b_bccc_48c8_81e5_94cae21943c6.slice/crio-8bd4127dfc5be27b94d98245586fd8e90d5bf0c2c225710aebca4e1d8053429a WatchSource:0}: Error finding container 8bd4127dfc5be27b94d98245586fd8e90d5bf0c2c225710aebca4e1d8053429a: Status 404 returned error can't find the container with id 8bd4127dfc5be27b94d98245586fd8e90d5bf0c2c225710aebca4e1d8053429a Sep 14 09:44:31.652982 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.652953 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:31.931724 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:31.931697 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-76v9c"] Sep 14 09:44:31.934088 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:44:31.934067 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2690c3_8c2f_4124_8575_e75a9a44d107.slice/crio-8cc56603ce0b9140b9a2d73e6c9e43874be8db3b3a600a8731006ec23b65bc59 WatchSource:0}: Error finding container 8cc56603ce0b9140b9a2d73e6c9e43874be8db3b3a600a8731006ec23b65bc59: Status 404 returned error can't find the container with id 8cc56603ce0b9140b9a2d73e6c9e43874be8db3b3a600a8731006ec23b65bc59 Sep 14 09:44:32.104103 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:32.104076 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-m5gcv\"" Sep 14 09:44:32.179607 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:32.179493 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" event={"ID":"5c6eb03b-bccc-48c8-81e5-94cae21943c6","Type":"ContainerStarted","Data":"8bd4127dfc5be27b94d98245586fd8e90d5bf0c2c225710aebca4e1d8053429a"} Sep 14 09:44:32.180788 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:32.180751 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" event={"ID":"6a2690c3-8c2f-4124-8575-e75a9a44d107","Type":"ContainerStarted","Data":"8cc56603ce0b9140b9a2d73e6c9e43874be8db3b3a600a8731006ec23b65bc59"} Sep 14 09:44:32.493950 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:32.493919 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 14 09:44:32.512323 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:32.512295 2550 projected.go:291] Couldn't get configMap openshift-cluster-storage-operator/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Sep 14 09:44:32.512323 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:32.512331 2550 projected.go:196] Error preparing data for projected volume kube-api-access-kvzdh for pod openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7: failed to sync configmap cache: timed out waiting for the condition Sep 14 09:44:32.512544 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:32.512395 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d3ccc69-de69-4209-a388-ce7d1c1b45b9-kube-api-access-kvzdh podName:2d3ccc69-de69-4209-a388-ce7d1c1b45b9 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:33.012374502 +0000 UTC m=+89.707499789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kvzdh" (UniqueName: "kubernetes.io/projected/2d3ccc69-de69-4209-a388-ce7d1c1b45b9-kube-api-access-kvzdh") pod "volume-data-source-validator-86695b7cc7-n8hq7" (UID: "2d3ccc69-de69-4209-a388-ce7d1c1b45b9") : failed to sync configmap cache: timed out waiting for the condition Sep 14 09:44:32.749260 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:32.749178 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 14 09:44:33.082031 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:33.081949 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvzdh\" (UniqueName: \"kubernetes.io/projected/2d3ccc69-de69-4209-a388-ce7d1c1b45b9-kube-api-access-kvzdh\") pod \"volume-data-source-validator-86695b7cc7-n8hq7\" (UID: \"2d3ccc69-de69-4209-a388-ce7d1c1b45b9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7" Sep 14 09:44:33.084423 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:33.084395 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvzdh\" (UniqueName: \"kubernetes.io/projected/2d3ccc69-de69-4209-a388-ce7d1c1b45b9-kube-api-access-kvzdh\") pod \"volume-data-source-validator-86695b7cc7-n8hq7\" (UID: \"2d3ccc69-de69-4209-a388-ce7d1c1b45b9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7" Sep 14 09:44:33.325334 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:33.325304 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7" Sep 14 09:44:34.188911 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:34.188874 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" event={"ID":"5c6eb03b-bccc-48c8-81e5-94cae21943c6","Type":"ContainerStarted","Data":"4cb4164684de22b5886dc96c4809aae1479718d4555be25f387b3db895281da2"} Sep 14 09:44:34.191222 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:34.191190 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" event={"ID":"6a2690c3-8c2f-4124-8575-e75a9a44d107","Type":"ContainerStarted","Data":"fa2c0d7fa016abef8292d335740ae3436d19062a80d610ec5f68a6a98d7a4bdc"} Sep 14 09:44:34.191480 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:34.191460 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:34.192914 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:34.192891 2550 patch_prober.go:28] interesting pod/console-operator-696884fb9c-76v9c container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 14 09:44:34.193027 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:34.192929 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" podUID="6a2690c3-8c2f-4124-8575-e75a9a44d107" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 14 09:44:34.262477 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:34.262423 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" podStartSLOduration=1.82141097 podStartE2EDuration="4.262405596s" totalImagesPullingTime="2.440994626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:34.215554209 +0000 UTC m=+90.910679514" watchObservedRunningTime="2026-09-14 09:44:34.262405596 +0000 UTC m=+90.957530902" Sep 14 09:44:34.363846 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:34.363753 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" podStartSLOduration=2.200943559 podStartE2EDuration="4.363720185s" totalImagesPullingTime="2.162776626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:34.263062681 +0000 UTC m=+90.958187999" watchObservedRunningTime="2026-09-14 09:44:34.363720185 +0000 UTC m=+91.058845493" Sep 14 09:44:34.364264 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:34.363868 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7"] Sep 14 09:44:34.366496 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:44:34.366469 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3ccc69_de69_4209_a388_ce7d1c1b45b9.slice/crio-52bf1193815d4519e91975d4592f6f413b934499f8c812e8e31e485ddf100799 WatchSource:0}: Error finding container 52bf1193815d4519e91975d4592f6f413b934499f8c812e8e31e485ddf100799: Status 404 returned error can't find the container with id 52bf1193815d4519e91975d4592f6f413b934499f8c812e8e31e485ddf100799 Sep 14 09:44:35.195826 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.195785 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7" event={"ID":"2d3ccc69-de69-4209-a388-ce7d1c1b45b9","Type":"ContainerStarted","Data":"52bf1193815d4519e91975d4592f6f413b934499f8c812e8e31e485ddf100799"} Sep 14 09:44:35.197648 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.197609 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/0.log" Sep 14 09:44:35.197780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.197668 2550 generic.go:356] "Generic (PLEG): container finished" podID="6a2690c3-8c2f-4124-8575-e75a9a44d107" containerID="fa2c0d7fa016abef8292d335740ae3436d19062a80d610ec5f68a6a98d7a4bdc" exitCode=255 Sep 14 09:44:35.197780 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.197759 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" event={"ID":"6a2690c3-8c2f-4124-8575-e75a9a44d107","Type":"ContainerDied","Data":"fa2c0d7fa016abef8292d335740ae3436d19062a80d610ec5f68a6a98d7a4bdc"} Sep 14 09:44:35.197981 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.197964 2550 scope.go:117] "RemoveContainer" containerID="fa2c0d7fa016abef8292d335740ae3436d19062a80d610ec5f68a6a98d7a4bdc" Sep 14 09:44:35.577237 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.577202 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tln2x"] Sep 14 09:44:35.580248 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.580228 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.582648 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.582614 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 14 09:44:35.582911 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.582892 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 14 09:44:35.583748 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.583712 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-22v7l\"" Sep 14 09:44:35.583841 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.583777 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 14 09:44:35.583841 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.583777 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 14 09:44:35.594731 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.594712 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tln2x"] Sep 14 09:44:35.699776 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.699740 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.699776 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.699778 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0607799d-9921-4e2d-ad78-c4816e060358-data-volume\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.699981 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.699800 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0607799d-9921-4e2d-ad78-c4816e060358-crio-socket\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.699981 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.699815 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.699981 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.699829 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5dnd\" (UniqueName: \"kubernetes.io/projected/0607799d-9921-4e2d-ad78-c4816e060358-kube-api-access-l5dnd\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.800187 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.800146 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.800187 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.800189 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0607799d-9921-4e2d-ad78-c4816e060358-data-volume\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.800391 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.800227 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0607799d-9921-4e2d-ad78-c4816e060358-crio-socket\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.800391 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.800251 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.800391 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.800274 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5dnd\" (UniqueName: \"kubernetes.io/projected/0607799d-9921-4e2d-ad78-c4816e060358-kube-api-access-l5dnd\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.800391 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:35.800310 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 09:44:35.800391 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.800363 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0607799d-9921-4e2d-ad78-c4816e060358-crio-socket\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.800554 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:35.800370 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 09:44:35.800554 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:35.800378 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls podName:0607799d-9921-4e2d-ad78-c4816e060358 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:36.300359478 +0000 UTC m=+92.995484774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tln2x" (UID: "0607799d-9921-4e2d-ad78-c4816e060358") : secret "insights-runtime-extractor-tls" not found Sep 14 09:44:35.800554 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:35.800470 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm podName:0607799d-9921-4e2d-ad78-c4816e060358 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:36.300450468 +0000 UTC m=+92.995575754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tln2x" (UID: "0607799d-9921-4e2d-ad78-c4816e060358") : configmap "kube-rbac-proxy" not found Sep 14 09:44:35.800689 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.800628 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0607799d-9921-4e2d-ad78-c4816e060358-data-volume\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:35.809312 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:35.809293 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5dnd\" (UniqueName: \"kubernetes.io/projected/0607799d-9921-4e2d-ad78-c4816e060358-kube-api-access-l5dnd\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:36.202244 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:36.202202 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7" event={"ID":"2d3ccc69-de69-4209-a388-ce7d1c1b45b9","Type":"ContainerStarted","Data":"6d7d5cf3323a03201b20bb1503904c8b8e4bd064793e52f39923c9659e7445ce"} Sep 14 09:44:36.203749 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:36.203730 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 09:44:36.204121 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:36.204107 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/0.log" Sep 14 09:44:36.204167 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:36.204138 2550 generic.go:356] "Generic (PLEG): container finished" podID="6a2690c3-8c2f-4124-8575-e75a9a44d107" containerID="6ba8d8513bf37e04d2a1983a66252e514e5d3e8e4195815d676dae6efd6176d2" exitCode=255 Sep 14 09:44:36.204208 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:36.204170 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" event={"ID":"6a2690c3-8c2f-4124-8575-e75a9a44d107","Type":"ContainerDied","Data":"6ba8d8513bf37e04d2a1983a66252e514e5d3e8e4195815d676dae6efd6176d2"} Sep 14 09:44:36.204240 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:36.204213 2550 scope.go:117] "RemoveContainer" containerID="fa2c0d7fa016abef8292d335740ae3436d19062a80d610ec5f68a6a98d7a4bdc" Sep 14 09:44:36.204384 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:36.204356 2550 scope.go:117] "RemoveContainer" containerID="6ba8d8513bf37e04d2a1983a66252e514e5d3e8e4195815d676dae6efd6176d2" Sep 14 09:44:36.204538 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:36.204517 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-76v9c_openshift-console-operator(6a2690c3-8c2f-4124-8575-e75a9a44d107)\"" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" podUID="6a2690c3-8c2f-4124-8575-e75a9a44d107" Sep 14 09:44:36.220195 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:36.220159 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n8hq7" podStartSLOduration=4.232398279 podStartE2EDuration="5.220144214s" totalImagesPullingTime="987.745935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:36.219911684 +0000 UTC m=+92.915037012" watchObservedRunningTime="2026-09-14 09:44:36.220144214 +0000 UTC m=+92.915269519" Sep 14 09:44:36.303335 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:36.303310 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:36.303436 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:36.303370 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:36.303498 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:36.303442 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 09:44:36.303498 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:36.303495 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls podName:0607799d-9921-4e2d-ad78-c4816e060358 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:37.303479897 +0000 UTC m=+93.998605180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tln2x" (UID: "0607799d-9921-4e2d-ad78-c4816e060358") : secret "insights-runtime-extractor-tls" not found Sep 14 09:44:36.303607 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:36.303529 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 09:44:36.303607 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:36.303593 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm podName:0607799d-9921-4e2d-ad78-c4816e060358 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:37.303576197 +0000 UTC m=+93.998701485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tln2x" (UID: "0607799d-9921-4e2d-ad78-c4816e060358") : configmap "kube-rbac-proxy" not found Sep 14 09:44:37.208374 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.208347 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 09:44:37.208760 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.208704 2550 scope.go:117] "RemoveContainer" containerID="6ba8d8513bf37e04d2a1983a66252e514e5d3e8e4195815d676dae6efd6176d2" Sep 14 09:44:37.208870 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:37.208851 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-76v9c_openshift-console-operator(6a2690c3-8c2f-4124-8575-e75a9a44d107)\"" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" podUID="6a2690c3-8c2f-4124-8575-e75a9a44d107" Sep 14 09:44:37.309998 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.309972 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:37.310174 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:37.310105 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 09:44:37.310174 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:37.310157 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm podName:0607799d-9921-4e2d-ad78-c4816e060358 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:39.310139502 +0000 UTC m=+96.005264798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tln2x" (UID: "0607799d-9921-4e2d-ad78-c4816e060358") : configmap "kube-rbac-proxy" not found Sep 14 09:44:37.310321 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.310192 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:37.310382 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:37.310320 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 09:44:37.310435 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:37.310381 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls podName:0607799d-9921-4e2d-ad78-c4816e060358 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:39.310364545 +0000 UTC m=+96.005489842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tln2x" (UID: "0607799d-9921-4e2d-ad78-c4816e060358") : secret "insights-runtime-extractor-tls" not found Sep 14 09:44:37.976226 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.976192 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-f6wnp"] Sep 14 09:44:37.980332 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.980312 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:37.983072 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.983053 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6x9vq\"" Sep 14 09:44:37.983176 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.983122 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 14 09:44:37.984343 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.984328 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 14 09:44:37.984418 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.984396 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 14 09:44:37.984747 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.984734 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 14 09:44:37.996043 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:37.996017 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-f6wnp"] Sep 14 09:44:38.019494 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.019467 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/feda6e9e-b311-47b3-b928-47c3891bc884-signing-cabundle\") pod \"service-ca-568869fdc5-f6wnp\" (UID: \"feda6e9e-b311-47b3-b928-47c3891bc884\") " pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:38.019585 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.019507 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/feda6e9e-b311-47b3-b928-47c3891bc884-signing-key\") pod \"service-ca-568869fdc5-f6wnp\" (UID: \"feda6e9e-b311-47b3-b928-47c3891bc884\") " pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:38.019585 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.019575 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xds27\" (UniqueName: \"kubernetes.io/projected/feda6e9e-b311-47b3-b928-47c3891bc884-kube-api-access-xds27\") pod \"service-ca-568869fdc5-f6wnp\" (UID: \"feda6e9e-b311-47b3-b928-47c3891bc884\") " pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:38.120192 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.120163 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/feda6e9e-b311-47b3-b928-47c3891bc884-signing-cabundle\") pod \"service-ca-568869fdc5-f6wnp\" (UID: \"feda6e9e-b311-47b3-b928-47c3891bc884\") " pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:38.120309 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.120203 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/feda6e9e-b311-47b3-b928-47c3891bc884-signing-key\") pod \"service-ca-568869fdc5-f6wnp\" (UID: \"feda6e9e-b311-47b3-b928-47c3891bc884\") " pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:38.120309 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.120230 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xds27\" (UniqueName: \"kubernetes.io/projected/feda6e9e-b311-47b3-b928-47c3891bc884-kube-api-access-xds27\") pod \"service-ca-568869fdc5-f6wnp\" (UID: \"feda6e9e-b311-47b3-b928-47c3891bc884\") " pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:38.120826 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.120807 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/feda6e9e-b311-47b3-b928-47c3891bc884-signing-cabundle\") pod \"service-ca-568869fdc5-f6wnp\" (UID: \"feda6e9e-b311-47b3-b928-47c3891bc884\") " pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:38.122445 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.122425 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/feda6e9e-b311-47b3-b928-47c3891bc884-signing-key\") pod \"service-ca-568869fdc5-f6wnp\" (UID: \"feda6e9e-b311-47b3-b928-47c3891bc884\") " pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:38.134200 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.134172 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xds27\" (UniqueName: \"kubernetes.io/projected/feda6e9e-b311-47b3-b928-47c3891bc884-kube-api-access-xds27\") pod \"service-ca-568869fdc5-f6wnp\" (UID: \"feda6e9e-b311-47b3-b928-47c3891bc884\") " pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:38.135108 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.135090 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4htsc_eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed/dns-node-resolver/0.log" Sep 14 09:44:38.288815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.288736 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" Sep 14 09:44:38.609997 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:38.609926 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-f6wnp"] Sep 14 09:44:38.612576 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:44:38.612544 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeda6e9e_b311_47b3_b928_47c3891bc884.slice/crio-13c11567ae4d1c41daf8257409b5bf0c9796f1c3e18c03278702b34908a6e4df WatchSource:0}: Error finding container 13c11567ae4d1c41daf8257409b5bf0c9796f1c3e18c03278702b34908a6e4df: Status 404 returned error can't find the container with id 13c11567ae4d1c41daf8257409b5bf0c9796f1c3e18c03278702b34908a6e4df Sep 14 09:44:39.218391 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:39.218353 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" event={"ID":"feda6e9e-b311-47b3-b928-47c3891bc884","Type":"ContainerStarted","Data":"12c1d37fe6cd84985165ec123b1da2f599e26887b46355037d900f2017a85087"} Sep 14 09:44:39.218554 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:39.218414 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" event={"ID":"feda6e9e-b311-47b3-b928-47c3891bc884","Type":"ContainerStarted","Data":"13c11567ae4d1c41daf8257409b5bf0c9796f1c3e18c03278702b34908a6e4df"} Sep 14 09:44:39.243107 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:39.243060 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-f6wnp" podStartSLOduration=2.243046904 podStartE2EDuration="2.243046904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:39.242491917 +0000 UTC m=+95.937617222" watchObservedRunningTime="2026-09-14 09:44:39.243046904 +0000 UTC m=+95.938172208" Sep 14 09:44:39.328884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:39.328854 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:39.329370 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:39.328942 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:39.329370 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:39.328984 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 09:44:39.329370 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:39.329042 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 09:44:39.329370 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:39.329052 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm podName:0607799d-9921-4e2d-ad78-c4816e060358 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:43.32903413 +0000 UTC m=+100.024159428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tln2x" (UID: "0607799d-9921-4e2d-ad78-c4816e060358") : configmap "kube-rbac-proxy" not found Sep 14 09:44:39.329370 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:39.329087 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls podName:0607799d-9921-4e2d-ad78-c4816e060358 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:43.32907381 +0000 UTC m=+100.024199112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tln2x" (UID: "0607799d-9921-4e2d-ad78-c4816e060358") : secret "insights-runtime-extractor-tls" not found Sep 14 09:44:39.530754 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:39.530683 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-phdx8_52a488f7-cd8d-4974-b6cb-5e293a21a2a6/node-ca/0.log" Sep 14 09:44:39.937823 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:39.937795 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-85622_53089d2e-4275-4a32-96e6-5800b3f71b08/serve-healthcheck-canary/0.log" Sep 14 09:44:41.042967 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:41.042929 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls\") pod \"image-registry-6956cc7fb6-pmxc7\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:44:41.043322 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:41.042990 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:44:41.043322 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:41.043080 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 09:44:41.043322 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:41.043099 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6956cc7fb6-pmxc7: secret "image-registry-tls" not found Sep 14 09:44:41.043322 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:41.043112 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 09:44:41.043322 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:41.043166 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls podName:8b94aee3-35c3-45ee-aa73-dd0083f07b78 nodeName:}" failed. No retries permitted until 2026-09-14 09:45:45.043143299 +0000 UTC m=+161.738268585 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls") pod "image-registry-6956cc7fb6-pmxc7" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78") : secret "image-registry-tls" not found Sep 14 09:44:41.043322 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:41.043183 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls podName:a7e4ee1c-60de-43ff-9b68-507c7ff58bdb nodeName:}" failed. No retries permitted until 2026-09-14 09:45:45.043173865 +0000 UTC m=+161.738299152 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls") pod "dns-default-q5n48" (UID: "a7e4ee1c-60de-43ff-9b68-507c7ff58bdb") : secret "dns-default-metrics-tls" not found Sep 14 09:44:41.653101 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:41.653067 2550 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:41.653425 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:41.653411 2550 scope.go:117] "RemoveContainer" containerID="6ba8d8513bf37e04d2a1983a66252e514e5d3e8e4195815d676dae6efd6176d2" Sep 14 09:44:41.653567 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:41.653550 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-76v9c_openshift-console-operator(6a2690c3-8c2f-4124-8575-e75a9a44d107)\"" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" podUID="6a2690c3-8c2f-4124-8575-e75a9a44d107" Sep 14 09:44:43.360427 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:43.360387 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:43.360935 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:43.360512 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:43.360935 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:43.360538 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 09:44:43.360935 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:43.360658 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm podName:0607799d-9921-4e2d-ad78-c4816e060358 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:51.360619598 +0000 UTC m=+108.055744897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tln2x" (UID: "0607799d-9921-4e2d-ad78-c4816e060358") : configmap "kube-rbac-proxy" not found Sep 14 09:44:43.360935 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:43.360718 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 09:44:43.360935 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:43.360792 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls podName:0607799d-9921-4e2d-ad78-c4816e060358 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:51.36077493 +0000 UTC m=+108.055900217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tln2x" (UID: "0607799d-9921-4e2d-ad78-c4816e060358") : secret "insights-runtime-extractor-tls" not found Sep 14 09:44:44.191933 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:44.191896 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:44.192243 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:44.192230 2550 scope.go:117] "RemoveContainer" containerID="6ba8d8513bf37e04d2a1983a66252e514e5d3e8e4195815d676dae6efd6176d2" Sep 14 09:44:44.192386 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:44:44.192367 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-76v9c_openshift-console-operator(6a2690c3-8c2f-4124-8575-e75a9a44d107)\"" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" podUID="6a2690c3-8c2f-4124-8575-e75a9a44d107" Sep 14 09:44:46.141972 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:46.141943 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7hkpf" Sep 14 09:44:51.421771 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:51.421733 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:51.422315 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:51.421816 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:51.422315 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:51.422200 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0607799d-9921-4e2d-ad78-c4816e060358-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:51.424020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:51.423997 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0607799d-9921-4e2d-ad78-c4816e060358-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tln2x\" (UID: \"0607799d-9921-4e2d-ad78-c4816e060358\") " pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:51.488715 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:51.488691 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tln2x" Sep 14 09:44:51.761134 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:51.761108 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tln2x"] Sep 14 09:44:51.763535 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:44:51.763495 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0607799d_9921_4e2d_ad78_c4816e060358.slice/crio-04e79b8964cfb25171022925498bc661a8c79e6b63e130d1ef99cd1786564322 WatchSource:0}: Error finding container 04e79b8964cfb25171022925498bc661a8c79e6b63e130d1ef99cd1786564322: Status 404 returned error can't find the container with id 04e79b8964cfb25171022925498bc661a8c79e6b63e130d1ef99cd1786564322 Sep 14 09:44:52.265534 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:52.265504 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tln2x" event={"ID":"0607799d-9921-4e2d-ad78-c4816e060358","Type":"ContainerStarted","Data":"08cc0e2164b8fd8d1f8785e16a0d3d65613a5d05936f8b78b6b122bd2f6d66ec"} Sep 14 09:44:52.265687 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:52.265557 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tln2x" event={"ID":"0607799d-9921-4e2d-ad78-c4816e060358","Type":"ContainerStarted","Data":"04e79b8964cfb25171022925498bc661a8c79e6b63e130d1ef99cd1786564322"} Sep 14 09:44:53.271109 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:53.271071 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tln2x" event={"ID":"0607799d-9921-4e2d-ad78-c4816e060358","Type":"ContainerStarted","Data":"478fc40b226cc7e6a7b77906d504e5eea37d966a01d81efa8e42d098598f63ac"} Sep 14 09:44:54.275783 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:54.275703 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tln2x" event={"ID":"0607799d-9921-4e2d-ad78-c4816e060358","Type":"ContainerStarted","Data":"3a71a5fd1813ecc177a4eba912c7d59ee0718a4b32de85e4ec9b05b7abf746f0"} Sep 14 09:44:54.298118 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:54.298073 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tln2x" podStartSLOduration=17.182597073 podStartE2EDuration="19.298059624s" totalImagesPullingTime="2.115462551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:44:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:54.296364321 +0000 UTC m=+110.991489629" watchObservedRunningTime="2026-09-14 09:44:54.298059624 +0000 UTC m=+110.993184960" Sep 14 09:44:55.818949 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:55.818916 2550 scope.go:117] "RemoveContainer" containerID="6ba8d8513bf37e04d2a1983a66252e514e5d3e8e4195815d676dae6efd6176d2" Sep 14 09:44:56.284372 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:56.284345 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 09:44:56.284567 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:56.284390 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" event={"ID":"6a2690c3-8c2f-4124-8575-e75a9a44d107","Type":"ContainerStarted","Data":"5beb1275dc951b908ff7bef7a4540108fcf54bb0b61bccc235842df5a6e790ca"} Sep 14 09:44:56.284681 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:56.284661 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:44:56.448734 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:44:56.448702 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-76v9c" Sep 14 09:45:00.416492 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.416462 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr"] Sep 14 09:45:00.418461 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.418438 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.421967 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.421941 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 14 09:45:00.423366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.423105 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 14 09:45:00.423366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.423176 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 14 09:45:00.423366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.423232 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 14 09:45:00.432502 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.432473 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr"] Sep 14 09:45:00.478259 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.478233 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-6dhlj"] Sep 14 09:45:00.480576 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.480558 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-6dhlj" Sep 14 09:45:00.483898 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.483876 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-m79hj\"" Sep 14 09:45:00.485095 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.485074 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 14 09:45:00.499337 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.499320 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 14 09:45:00.513368 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.513344 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-6dhlj"] Sep 14 09:45:00.589830 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.589796 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xlt\" (UniqueName: \"kubernetes.io/projected/9ca5a196-fcb6-47d1-aef4-24eea70254c6-kube-api-access-v7xlt\") pod \"downloads-76f99584f4-6dhlj\" (UID: \"9ca5a196-fcb6-47d1-aef4-24eea70254c6\") " pod="openshift-console/downloads-76f99584f4-6dhlj" Sep 14 09:45:00.589984 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.589837 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1-tmp\") pod \"klusterlet-addon-workmgr-5db4fdfdb7-nbvfr\" (UID: \"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.589984 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.589900 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bmws\" (UniqueName: \"kubernetes.io/projected/b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1-kube-api-access-5bmws\") pod \"klusterlet-addon-workmgr-5db4fdfdb7-nbvfr\" (UID: \"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.589984 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.589941 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1-klusterlet-config\") pod \"klusterlet-addon-workmgr-5db4fdfdb7-nbvfr\" (UID: \"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.690343 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.690279 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xlt\" (UniqueName: \"kubernetes.io/projected/9ca5a196-fcb6-47d1-aef4-24eea70254c6-kube-api-access-v7xlt\") pod \"downloads-76f99584f4-6dhlj\" (UID: \"9ca5a196-fcb6-47d1-aef4-24eea70254c6\") " pod="openshift-console/downloads-76f99584f4-6dhlj" Sep 14 09:45:00.690343 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.690312 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1-tmp\") pod \"klusterlet-addon-workmgr-5db4fdfdb7-nbvfr\" (UID: \"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.690343 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.690335 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bmws\" (UniqueName: \"kubernetes.io/projected/b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1-kube-api-access-5bmws\") pod \"klusterlet-addon-workmgr-5db4fdfdb7-nbvfr\" (UID: \"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.690513 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.690361 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1-klusterlet-config\") pod \"klusterlet-addon-workmgr-5db4fdfdb7-nbvfr\" (UID: \"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.690791 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.690764 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1-tmp\") pod \"klusterlet-addon-workmgr-5db4fdfdb7-nbvfr\" (UID: \"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.692977 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.692952 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1-klusterlet-config\") pod \"klusterlet-addon-workmgr-5db4fdfdb7-nbvfr\" (UID: \"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.701893 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.701871 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bmws\" (UniqueName: \"kubernetes.io/projected/b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1-kube-api-access-5bmws\") pod \"klusterlet-addon-workmgr-5db4fdfdb7-nbvfr\" (UID: \"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.702431 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.702414 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xlt\" (UniqueName: \"kubernetes.io/projected/9ca5a196-fcb6-47d1-aef4-24eea70254c6-kube-api-access-v7xlt\") pod \"downloads-76f99584f4-6dhlj\" (UID: \"9ca5a196-fcb6-47d1-aef4-24eea70254c6\") " pod="openshift-console/downloads-76f99584f4-6dhlj" Sep 14 09:45:00.729089 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.729066 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:00.788850 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:00.788823 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-6dhlj" Sep 14 09:45:01.025491 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:01.025467 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr"] Sep 14 09:45:01.027977 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:45:01.027953 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ffb557_ebe8_40c6_a8bf_b029cb5e69c1.slice/crio-2410fc6811f84fcb695864b5ba5bef5e256e1a29703e67cfe381bfc10a4054db WatchSource:0}: Error finding container 2410fc6811f84fcb695864b5ba5bef5e256e1a29703e67cfe381bfc10a4054db: Status 404 returned error can't find the container with id 2410fc6811f84fcb695864b5ba5bef5e256e1a29703e67cfe381bfc10a4054db Sep 14 09:45:01.068950 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:01.068914 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-6dhlj"] Sep 14 09:45:01.070823 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:45:01.070802 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca5a196_fcb6_47d1_aef4_24eea70254c6.slice/crio-ff006fced1785b18fdd8dc1d7bd3bb16e0332320f4e11d002741b10d7035c0d6 WatchSource:0}: Error finding container ff006fced1785b18fdd8dc1d7bd3bb16e0332320f4e11d002741b10d7035c0d6: Status 404 returned error can't find the container with id ff006fced1785b18fdd8dc1d7bd3bb16e0332320f4e11d002741b10d7035c0d6 Sep 14 09:45:01.302790 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:01.302706 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" event={"ID":"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1","Type":"ContainerStarted","Data":"2410fc6811f84fcb695864b5ba5bef5e256e1a29703e67cfe381bfc10a4054db"} Sep 14 09:45:01.303698 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:01.303673 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-6dhlj" event={"ID":"9ca5a196-fcb6-47d1-aef4-24eea70254c6","Type":"ContainerStarted","Data":"ff006fced1785b18fdd8dc1d7bd3bb16e0332320f4e11d002741b10d7035c0d6"} Sep 14 09:45:05.397781 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.397748 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bdd99fc67-996zv"] Sep 14 09:45:05.402502 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.402480 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.405370 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.405192 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 14 09:45:05.406101 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.405828 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 14 09:45:05.406101 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.406034 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 14 09:45:05.406565 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.406420 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 14 09:45:05.406565 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.406460 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 14 09:45:05.406565 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.406486 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-stffz\"" Sep 14 09:45:05.418596 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.418570 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdd99fc67-996zv"] Sep 14 09:45:05.531362 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.531331 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-oauth-config\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.531534 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.531383 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-serving-cert\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.531534 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.531410 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-console-config\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.531534 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.531451 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lsp\" (UniqueName: \"kubernetes.io/projected/031f231a-7bda-4b8c-a595-3122a311443d-kube-api-access-p2lsp\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.531686 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.531535 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-service-ca\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.531686 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.531644 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-oauth-serving-cert\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.632709 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.632624 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-oauth-config\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.632709 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.632693 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-serving-cert\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.632908 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.632720 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-console-config\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.632908 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.632748 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lsp\" (UniqueName: \"kubernetes.io/projected/031f231a-7bda-4b8c-a595-3122a311443d-kube-api-access-p2lsp\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.632908 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.632775 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-service-ca\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.632908 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.632848 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-oauth-serving-cert\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.633528 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.633500 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-console-config\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.633528 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.633522 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-oauth-serving-cert\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.633720 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.633655 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-service-ca\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.635482 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.635460 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-serving-cert\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.635851 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.635830 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-oauth-config\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.644073 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.644052 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lsp\" (UniqueName: \"kubernetes.io/projected/031f231a-7bda-4b8c-a595-3122a311443d-kube-api-access-p2lsp\") pod \"console-5bdd99fc67-996zv\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:05.730498 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:05.730464 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:06.053159 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.053124 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdd99fc67-996zv"] Sep 14 09:45:06.056808 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:45:06.056780 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031f231a_7bda_4b8c_a595_3122a311443d.slice/crio-719d519c44228924eaa0a4e818fc625ccf94bf73c596eaf0e74587d973d84aa4 WatchSource:0}: Error finding container 719d519c44228924eaa0a4e818fc625ccf94bf73c596eaf0e74587d973d84aa4: Status 404 returned error can't find the container with id 719d519c44228924eaa0a4e818fc625ccf94bf73c596eaf0e74587d973d84aa4 Sep 14 09:45:06.224090 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.224055 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj"] Sep 14 09:45:06.249671 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.249623 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj"] Sep 14 09:45:06.249813 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.249730 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj" Sep 14 09:45:06.252222 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.252198 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-glgx2\"" Sep 14 09:45:06.252352 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.252196 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 14 09:45:06.329872 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.329792 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdd99fc67-996zv" event={"ID":"031f231a-7bda-4b8c-a595-3122a311443d","Type":"ContainerStarted","Data":"719d519c44228924eaa0a4e818fc625ccf94bf73c596eaf0e74587d973d84aa4"} Sep 14 09:45:06.331619 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.331582 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" event={"ID":"b2ffb557-ebe8-40c6-a8bf-b029cb5e69c1","Type":"ContainerStarted","Data":"5a0d132a5a20e4747f2e096afa8a4179d34b9c9feb70c9d0d77900ca626b1e77"} Sep 14 09:45:06.331931 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.331912 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:06.333529 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.333503 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" Sep 14 09:45:06.339210 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.339189 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94a2a535-c7dd-40f8-a620-67aedbbabbe3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-48lrj\" (UID: \"94a2a535-c7dd-40f8-a620-67aedbbabbe3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj" Sep 14 09:45:06.349952 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.349918 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5db4fdfdb7-nbvfr" podStartSLOduration=1.995296444 podStartE2EDuration="6.349905051s" totalImagesPullingTime="4.354608607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:06.348557236 +0000 UTC m=+123.043682542" watchObservedRunningTime="2026-09-14 09:45:06.349905051 +0000 UTC m=+123.045030357" Sep 14 09:45:06.440511 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.440486 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94a2a535-c7dd-40f8-a620-67aedbbabbe3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-48lrj\" (UID: \"94a2a535-c7dd-40f8-a620-67aedbbabbe3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj" Sep 14 09:45:06.443373 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.443341 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94a2a535-c7dd-40f8-a620-67aedbbabbe3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-48lrj\" (UID: \"94a2a535-c7dd-40f8-a620-67aedbbabbe3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj" Sep 14 09:45:06.561168 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.561127 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj" Sep 14 09:45:06.850046 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:06.850023 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj"] Sep 14 09:45:06.853257 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:45:06.852887 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a2a535_c7dd_40f8_a620_67aedbbabbe3.slice/crio-d0f64f810f1a78a0b0b4cfe51805beee3e11c2e69bb12cdf0c94d162cf0e7a59 WatchSource:0}: Error finding container d0f64f810f1a78a0b0b4cfe51805beee3e11c2e69bb12cdf0c94d162cf0e7a59: Status 404 returned error can't find the container with id d0f64f810f1a78a0b0b4cfe51805beee3e11c2e69bb12cdf0c94d162cf0e7a59 Sep 14 09:45:07.337784 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:07.337709 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj" event={"ID":"94a2a535-c7dd-40f8-a620-67aedbbabbe3","Type":"ContainerStarted","Data":"d0f64f810f1a78a0b0b4cfe51805beee3e11c2e69bb12cdf0c94d162cf0e7a59"} Sep 14 09:45:10.350646 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:10.350606 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdd99fc67-996zv" event={"ID":"031f231a-7bda-4b8c-a595-3122a311443d","Type":"ContainerStarted","Data":"921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39"} Sep 14 09:45:10.352413 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:10.352384 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj" event={"ID":"94a2a535-c7dd-40f8-a620-67aedbbabbe3","Type":"ContainerStarted","Data":"36ee70ec093b06cf5c0035e44f8d46ead9d8ee239f41b74a506f8ed7d913ceee"} Sep 14 09:45:10.352561 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:10.352537 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj" Sep 14 09:45:10.357751 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:10.357725 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj" Sep 14 09:45:10.371692 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:10.371647 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bdd99fc67-996zv" podStartSLOduration=2.066848065 podStartE2EDuration="5.371614731s" totalImagesPullingTime="3.304766666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:10.370096596 +0000 UTC m=+127.065221901" watchObservedRunningTime="2026-09-14 09:45:10.371614731 +0000 UTC m=+127.066740036" Sep 14 09:45:10.389148 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:10.389093 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-48lrj" podStartSLOduration=1.8882581859999998 podStartE2EDuration="4.389073785s" totalImagesPullingTime="2.500815599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:10.38808891 +0000 UTC m=+127.083214216" watchObservedRunningTime="2026-09-14 09:45:10.389073785 +0000 UTC m=+127.084199091" Sep 14 09:45:13.602375 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:13.602339 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:45:13.604987 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:13.604955 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b05f60-7a5d-498c-9e7b-e0bc98d0d495-metrics-certs\") pod \"network-metrics-daemon-lnvrp\" (UID: \"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495\") " pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:45:13.835135 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:13.835105 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ctbn\"" Sep 14 09:45:13.843040 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:13.843011 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnvrp" Sep 14 09:45:15.700009 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.699480 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7j5mh"] Sep 14 09:45:15.704066 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.704023 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.707705 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.707357 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 14 09:45:15.708998 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.708500 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ghd85\"" Sep 14 09:45:15.708998 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.708664 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 14 09:45:15.708998 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.708866 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 14 09:45:15.708998 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.708889 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 14 09:45:15.709280 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.709114 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 14 09:45:15.709361 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.709308 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 14 09:45:15.716057 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.715902 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/779e293c-f1ff-48c9-9f68-bfaae2520712-metrics-client-ca\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.716057 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.715955 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.716057 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.715983 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87dlz\" (UniqueName: \"kubernetes.io/projected/779e293c-f1ff-48c9-9f68-bfaae2520712-kube-api-access-87dlz\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.716057 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.716029 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/779e293c-f1ff-48c9-9f68-bfaae2520712-sys\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.716295 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.716064 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-textfile\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.716295 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.716101 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-tls\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.716295 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.716129 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.716295 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.716171 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/779e293c-f1ff-48c9-9f68-bfaae2520712-root\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.716295 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.716194 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-wtmp\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.730614 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.730586 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:15.730739 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.730685 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:15.731998 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.731975 2550 patch_prober.go:28] interesting pod/console-5bdd99fc67-996zv container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body= Sep 14 09:45:15.732096 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.732022 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5bdd99fc67-996zv" podUID="031f231a-7bda-4b8c-a595-3122a311443d" containerName="console" probeResult="failure" output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused" Sep 14 09:45:15.816987 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.816952 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/779e293c-f1ff-48c9-9f68-bfaae2520712-sys\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817180 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817008 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-textfile\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817180 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817073 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/779e293c-f1ff-48c9-9f68-bfaae2520712-sys\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817180 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817154 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-tls\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817184 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817229 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/779e293c-f1ff-48c9-9f68-bfaae2520712-root\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817253 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-wtmp\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817325 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/779e293c-f1ff-48c9-9f68-bfaae2520712-metrics-client-ca\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817335 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/779e293c-f1ff-48c9-9f68-bfaae2520712-root\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817335 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-textfile\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817674 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817481 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-wtmp\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817674 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817527 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817674 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:45:15.817561 2550 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 14 09:45:15.817674 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:45:15.817665 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-tls podName:779e293c-f1ff-48c9-9f68-bfaae2520712 nodeName:}" failed. No retries permitted until 2026-09-14 09:45:16.317618859 +0000 UTC m=+133.012744145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-tls") pod "node-exporter-7j5mh" (UID: "779e293c-f1ff-48c9-9f68-bfaae2520712") : secret "node-exporter-tls" not found Sep 14 09:45:15.817886 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817561 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87dlz\" (UniqueName: \"kubernetes.io/projected/779e293c-f1ff-48c9-9f68-bfaae2520712-kube-api-access-87dlz\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817950 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817883 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.817950 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.817894 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/779e293c-f1ff-48c9-9f68-bfaae2520712-metrics-client-ca\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.820160 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.820135 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:15.826864 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:15.826819 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87dlz\" (UniqueName: \"kubernetes.io/projected/779e293c-f1ff-48c9-9f68-bfaae2520712-kube-api-access-87dlz\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:16.323140 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.323104 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-tls\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:16.325736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.325707 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/779e293c-f1ff-48c9-9f68-bfaae2520712-node-exporter-tls\") pod \"node-exporter-7j5mh\" (UID: \"779e293c-f1ff-48c9-9f68-bfaae2520712\") " pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:16.617779 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.617692 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7j5mh" Sep 14 09:45:16.750606 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.750577 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 09:45:16.754349 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.754327 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.757047 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.756985 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 14 09:45:16.757047 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.756990 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 14 09:45:16.757047 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.757035 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 14 09:45:16.757266 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.756990 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 14 09:45:16.757319 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.757292 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 14 09:45:16.757371 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.757360 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 14 09:45:16.757483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.757461 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 14 09:45:16.757616 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.757568 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 14 09:45:16.757708 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.757651 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-982b5\"" Sep 14 09:45:16.757763 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.757703 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 14 09:45:16.774542 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.774520 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 09:45:16.826473 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826438 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826618 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826481 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggwqh\" (UniqueName: \"kubernetes.io/projected/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-kube-api-access-ggwqh\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826618 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826513 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826618 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826540 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826618 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826562 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826838 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826686 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-web-config\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826838 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826730 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826838 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826783 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-config-out\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826973 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826843 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826973 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826867 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826973 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826889 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826973 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826912 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.826973 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.826934 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928305 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928270 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-web-config\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928480 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928325 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928480 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928357 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-config-out\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928480 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928398 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928480 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928422 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928480 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928445 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928480 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928469 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928867 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928491 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928867 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928535 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928867 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928556 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggwqh\" (UniqueName: \"kubernetes.io/projected/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-kube-api-access-ggwqh\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928867 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928609 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928867 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928648 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.928867 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.928670 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.930266 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.929607 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.930266 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:45:16.929746 2550 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 14 09:45:16.930266 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:45:16.929818 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-main-tls podName:7e632b34-a7bd-4f48-8750-bfdc1eca6b9e nodeName:}" failed. No retries permitted until 2026-09-14 09:45:17.429800433 +0000 UTC m=+134.124925735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "7e632b34-a7bd-4f48-8750-bfdc1eca6b9e") : secret "alertmanager-main-tls" not found Sep 14 09:45:16.930947 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.930883 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.931660 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.931284 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.932531 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.932046 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.932531 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.932069 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-web-config\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.932531 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.932185 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.932531 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.932212 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.932531 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.932482 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.933514 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.933471 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-config-out\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.934473 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.934451 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.934556 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.934524 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:16.941921 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:16.941899 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggwqh\" (UniqueName: \"kubernetes.io/projected/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-kube-api-access-ggwqh\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:17.432660 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.432612 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:17.435329 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.435294 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e632b34-a7bd-4f48-8750-bfdc1eca6b9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:17.666125 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.666091 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 09:45:17.767444 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.767366 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb"] Sep 14 09:45:17.770980 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.770960 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.777739 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.777675 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 14 09:45:17.777739 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.777687 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 14 09:45:17.777919 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.777845 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 14 09:45:17.777989 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.777943 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-47nkp\"" Sep 14 09:45:17.778872 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.778660 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 14 09:45:17.778872 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.778712 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d9do61eb5vk5q\"" Sep 14 09:45:17.778872 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.778660 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 14 09:45:17.788984 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.788965 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb"] Sep 14 09:45:17.837014 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.836979 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-grpc-tls\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.837162 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.837036 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-tls\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.837162 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.837065 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.837162 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.837107 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.837321 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.837190 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.837321 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.837238 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feb0e927-0450-41ab-9427-93dfe0392abe-metrics-client-ca\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.837419 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.837320 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmksf\" (UniqueName: \"kubernetes.io/projected/feb0e927-0450-41ab-9427-93dfe0392abe-kube-api-access-qmksf\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.837419 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.837372 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.938623 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.938589 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.938818 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.938660 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feb0e927-0450-41ab-9427-93dfe0392abe-metrics-client-ca\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.938818 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.938706 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmksf\" (UniqueName: \"kubernetes.io/projected/feb0e927-0450-41ab-9427-93dfe0392abe-kube-api-access-qmksf\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.938818 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.938741 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.938971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.938894 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-grpc-tls\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.938971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.938963 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-tls\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.939074 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.938999 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.939074 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.939025 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.939628 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.939577 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feb0e927-0450-41ab-9427-93dfe0392abe-metrics-client-ca\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.941749 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.941727 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.941887 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.941728 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.941949 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.941931 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-tls\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.942458 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.942436 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.942458 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.942449 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-grpc-tls\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.942833 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.942816 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/feb0e927-0450-41ab-9427-93dfe0392abe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:17.948802 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:17.948776 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmksf\" (UniqueName: \"kubernetes.io/projected/feb0e927-0450-41ab-9427-93dfe0392abe-kube-api-access-qmksf\") pod \"thanos-querier-6cd55dd6f9-f5vrb\" (UID: \"feb0e927-0450-41ab-9427-93dfe0392abe\") " pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:18.081726 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:18.081626 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:19.353940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.353865 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lnvrp"] Sep 14 09:45:19.359530 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:45:19.359496 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b05f60_7a5d_498c_9e7b_e0bc98d0d495.slice/crio-7bc81c9fd324566bf4f95cf5a8bc1dc5e385831d745c4cdbc7ee5925a5fedef4 WatchSource:0}: Error finding container 7bc81c9fd324566bf4f95cf5a8bc1dc5e385831d745c4cdbc7ee5925a5fedef4: Status 404 returned error can't find the container with id 7bc81c9fd324566bf4f95cf5a8bc1dc5e385831d745c4cdbc7ee5925a5fedef4 Sep 14 09:45:19.375459 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.375428 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb"] Sep 14 09:45:19.376850 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:45:19.376791 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb0e927_0450_41ab_9427_93dfe0392abe.slice/crio-d18308a837a13cd66890aee285794d89fa0d4fa82bfc9d34a10e013489667149 WatchSource:0}: Error finding container d18308a837a13cd66890aee285794d89fa0d4fa82bfc9d34a10e013489667149: Status 404 returned error can't find the container with id d18308a837a13cd66890aee285794d89fa0d4fa82bfc9d34a10e013489667149 Sep 14 09:45:19.385903 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.385570 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 09:45:19.386907 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:45:19.386823 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e632b34_a7bd_4f48_8750_bfdc1eca6b9e.slice/crio-460e9142298510f9963affb83937ef38dab3ceccf645ec7aeb6d491b4a11bcd7 WatchSource:0}: Error finding container 460e9142298510f9963affb83937ef38dab3ceccf645ec7aeb6d491b4a11bcd7: Status 404 returned error can't find the container with id 460e9142298510f9963affb83937ef38dab3ceccf645ec7aeb6d491b4a11bcd7 Sep 14 09:45:19.390691 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.390587 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-6dhlj" event={"ID":"9ca5a196-fcb6-47d1-aef4-24eea70254c6","Type":"ContainerStarted","Data":"6007c65775102c02955a59d21d15087bb4ce89977c06c58b224d34e9e22441ad"} Sep 14 09:45:19.391040 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.390993 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-6dhlj" Sep 14 09:45:19.392226 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.392190 2550 patch_prober.go:28] interesting pod/downloads-76f99584f4-6dhlj container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.15:8080/\": dial tcp 10.132.0.15:8080: connect: connection refused" start-of-body= Sep 14 09:45:19.392334 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.392235 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-6dhlj" podUID="9ca5a196-fcb6-47d1-aef4-24eea70254c6" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.15:8080/\": dial tcp 10.132.0.15:8080: connect: connection refused" Sep 14 09:45:19.392800 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.392770 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" event={"ID":"feb0e927-0450-41ab-9427-93dfe0392abe","Type":"ContainerStarted","Data":"d18308a837a13cd66890aee285794d89fa0d4fa82bfc9d34a10e013489667149"} Sep 14 09:45:19.394469 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.394447 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnvrp" event={"ID":"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495","Type":"ContainerStarted","Data":"7bc81c9fd324566bf4f95cf5a8bc1dc5e385831d745c4cdbc7ee5925a5fedef4"} Sep 14 09:45:19.395971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.395886 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j5mh" event={"ID":"779e293c-f1ff-48c9-9f68-bfaae2520712","Type":"ContainerStarted","Data":"615c4a2c61e303a3088eee4418504bc5ef26cfb3303ea7fda807eeecc6745f84"} Sep 14 09:45:19.409395 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:19.409349 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-6dhlj" podStartSLOduration=1.284498965 podStartE2EDuration="19.409330875s" totalImagesPullingTime="18.12483191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:19.40827357 +0000 UTC m=+136.103398888" watchObservedRunningTime="2026-09-14 09:45:19.409330875 +0000 UTC m=+136.104456181" Sep 14 09:45:20.406363 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.404816 2550 generic.go:356] "Generic (PLEG): container finished" podID="779e293c-f1ff-48c9-9f68-bfaae2520712" containerID="cdcc30508076351fd3301d5732b4d97325b0418619c2a4f621a94c83ef175897" exitCode=0 Sep 14 09:45:20.406363 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.405552 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j5mh" event={"ID":"779e293c-f1ff-48c9-9f68-bfaae2520712","Type":"ContainerDied","Data":"cdcc30508076351fd3301d5732b4d97325b0418619c2a4f621a94c83ef175897"} Sep 14 09:45:20.412808 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.412713 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e","Type":"ContainerStarted","Data":"460e9142298510f9963affb83937ef38dab3ceccf645ec7aeb6d491b4a11bcd7"} Sep 14 09:45:20.418303 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.418281 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-6dhlj" Sep 14 09:45:20.463374 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.463318 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdd99fc67-996zv"] Sep 14 09:45:20.921382 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.921275 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5486df5f55-kj7bk"] Sep 14 09:45:20.943523 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.943452 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5486df5f55-kj7bk"] Sep 14 09:45:20.943753 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.943577 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:20.949302 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.949166 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 14 09:45:20.951004 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.950973 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 14 09:45:20.951320 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.951202 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 14 09:45:20.952373 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.951435 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xvt27\"" Sep 14 09:45:20.952373 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.951518 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 14 09:45:20.952992 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.952829 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 14 09:45:20.956272 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.956253 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 14 09:45:20.969261 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.969104 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-telemeter-client-tls\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:20.969261 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.969164 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-federate-client-tls\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:20.969261 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.969192 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3f931a3-7a96-44d7-beac-43811b45bb1a-metrics-client-ca\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:20.969261 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.969224 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f931a3-7a96-44d7-beac-43811b45bb1a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:20.969568 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.969374 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f931a3-7a96-44d7-beac-43811b45bb1a-serving-certs-ca-bundle\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:20.969568 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.969413 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-secret-telemeter-client\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:20.969568 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.969462 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:20.969568 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:20.969498 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k9d2\" (UniqueName: \"kubernetes.io/projected/b3f931a3-7a96-44d7-beac-43811b45bb1a-kube-api-access-7k9d2\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.071667 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.070385 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-telemeter-client-tls\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.071667 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.070451 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-federate-client-tls\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.071667 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.070477 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3f931a3-7a96-44d7-beac-43811b45bb1a-metrics-client-ca\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.071667 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.070507 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f931a3-7a96-44d7-beac-43811b45bb1a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.071667 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.070596 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f931a3-7a96-44d7-beac-43811b45bb1a-serving-certs-ca-bundle\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.071667 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.070625 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-secret-telemeter-client\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.071667 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.070673 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.071667 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.070710 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k9d2\" (UniqueName: \"kubernetes.io/projected/b3f931a3-7a96-44d7-beac-43811b45bb1a-kube-api-access-7k9d2\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.071667 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.071440 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3f931a3-7a96-44d7-beac-43811b45bb1a-metrics-client-ca\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.072250 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.072151 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f931a3-7a96-44d7-beac-43811b45bb1a-serving-certs-ca-bundle\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.072672 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.072611 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3f931a3-7a96-44d7-beac-43811b45bb1a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.076934 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.076902 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-federate-client-tls\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.077105 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.077052 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.077845 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.077798 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-telemeter-client-tls\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.080589 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.080546 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k9d2\" (UniqueName: \"kubernetes.io/projected/b3f931a3-7a96-44d7-beac-43811b45bb1a-kube-api-access-7k9d2\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.080736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.080624 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b3f931a3-7a96-44d7-beac-43811b45bb1a-secret-telemeter-client\") pod \"telemeter-client-5486df5f55-kj7bk\" (UID: \"b3f931a3-7a96-44d7-beac-43811b45bb1a\") " pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.259446 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.259292 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" Sep 14 09:45:21.425181 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.425110 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j5mh" event={"ID":"779e293c-f1ff-48c9-9f68-bfaae2520712","Type":"ContainerStarted","Data":"79fee4eba61e99e255e7a342b46f9695ae7c20183b1fe5e42d5032b66d4ec433"} Sep 14 09:45:21.425831 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:21.425688 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j5mh" event={"ID":"779e293c-f1ff-48c9-9f68-bfaae2520712","Type":"ContainerStarted","Data":"cb5014d2d9c5f9a736db95de75ad213e45853138c3dbc3f2119ecda36bd71710"} Sep 14 09:45:22.153126 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.153059 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7j5mh" podStartSLOduration=6.138673983 podStartE2EDuration="7.153036948s" totalImagesPullingTime="926.022156ms" totalInitContainerRuntime="88.340809ms" isStatefulPod=true podCreationTimestamp="2026-09-14 09:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:21.465547819 +0000 UTC m=+138.160673125" watchObservedRunningTime="2026-09-14 09:45:22.153036948 +0000 UTC m=+138.848162253" Sep 14 09:45:22.153737 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.153701 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 09:45:22.169417 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.169394 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.172236 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.172212 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 14 09:45:22.172381 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.172329 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 14 09:45:22.172961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.172806 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 14 09:45:22.172961 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.172807 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 14 09:45:22.173552 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.173528 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3hbl40kve9qg1\"" Sep 14 09:45:22.173826 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.173813 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 14 09:45:22.174001 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.173983 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 14 09:45:22.174157 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.174024 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 14 09:45:22.174303 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.174214 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 14 09:45:22.174476 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.174457 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 14 09:45:22.174712 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.174694 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cn65s\"" Sep 14 09:45:22.174969 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.174943 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 14 09:45:22.175195 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.175178 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 14 09:45:22.175393 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.175375 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 14 09:45:22.175393 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.175379 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 14 09:45:22.176945 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.176786 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 14 09:45:22.177469 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.177386 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 14 09:45:22.181024 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.180728 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 09:45:22.282321 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282119 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282321 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282173 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282321 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282201 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282330 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282379 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282402 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282453 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282481 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282505 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282559 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282593 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrtp\" (UniqueName: \"kubernetes.io/projected/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-kube-api-access-fbrtp\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282623 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-config\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282678 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282697 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.282736 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282729 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.283304 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282762 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.283304 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282786 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.283304 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282809 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.283304 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282894 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.283304 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.282975 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384288 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384247 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384497 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384324 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384497 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384358 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384497 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384384 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384497 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384409 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384497 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384442 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384497 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384479 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384511 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384530 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384581 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384614 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384657 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384706 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384732 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384755 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384782 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384809 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrtp\" (UniqueName: \"kubernetes.io/projected/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-kube-api-access-fbrtp\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.384848 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384835 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-config\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.385365 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384874 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.385365 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.384904 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.385365 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.385200 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.385691 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.385662 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.386153 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.386125 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.386714 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.386348 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.386829 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.386800 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.387689 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.387668 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.392210 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.391269 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.392210 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.391705 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.392210 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.392100 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.394184 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.392219 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.394504 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.394461 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.394713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.394659 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.396120 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.395094 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-config\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.396120 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.395263 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.396120 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.395733 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.396120 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.396009 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.396120 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.396012 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.396120 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.396080 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.396457 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.396249 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.398493 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.398447 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrtp\" (UniqueName: \"kubernetes.io/projected/f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36-kube-api-access-fbrtp\") pod \"prometheus-k8s-0\" (UID: \"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.485077 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.485014 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:22.964198 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.964166 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5486df5f55-kj7bk"] Sep 14 09:45:22.966246 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:45:22.966223 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f931a3_7a96_44d7_beac_43811b45bb1a.slice/crio-11c8cce64c1b0a31bc3aaa260017fba9177d6fcf80a0d443c774e3a3e3f42f7c WatchSource:0}: Error finding container 11c8cce64c1b0a31bc3aaa260017fba9177d6fcf80a0d443c774e3a3e3f42f7c: Status 404 returned error can't find the container with id 11c8cce64c1b0a31bc3aaa260017fba9177d6fcf80a0d443c774e3a3e3f42f7c Sep 14 09:45:22.971005 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:22.970912 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 09:45:22.973954 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:45:22.973927 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b4013d_ad1a_4d7b_8d41_66bfaadfbe36.slice/crio-306da401827b0eeb3354a4601d865e48241a0011ab79a5b8917fb60b270923bf WatchSource:0}: Error finding container 306da401827b0eeb3354a4601d865e48241a0011ab79a5b8917fb60b270923bf: Status 404 returned error can't find the container with id 306da401827b0eeb3354a4601d865e48241a0011ab79a5b8917fb60b270923bf Sep 14 09:45:23.435886 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.435851 2550 generic.go:356] "Generic (PLEG): container finished" podID="7e632b34-a7bd-4f48-8750-bfdc1eca6b9e" containerID="4b407aaadd52dcba60e5d99709a456311602108a2d08afed00ae09f2be3d1a3f" exitCode=0 Sep 14 09:45:23.436086 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.435939 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e","Type":"ContainerDied","Data":"4b407aaadd52dcba60e5d99709a456311602108a2d08afed00ae09f2be3d1a3f"} Sep 14 09:45:23.438936 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.438909 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" event={"ID":"feb0e927-0450-41ab-9427-93dfe0392abe","Type":"ContainerStarted","Data":"efc19a254fb04404e57412793e365c65166ac2fed0c96a71b0ac42c0e1f0a0a5"} Sep 14 09:45:23.439038 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.438977 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" event={"ID":"feb0e927-0450-41ab-9427-93dfe0392abe","Type":"ContainerStarted","Data":"78bae3a2e58880bf7edeef514a7ae5feb0180a44f9453e9633f425c9fa7101db"} Sep 14 09:45:23.439038 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.439023 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" event={"ID":"feb0e927-0450-41ab-9427-93dfe0392abe","Type":"ContainerStarted","Data":"cbd786198b3b22a4cb590ac523fb06620f07684758913dd6a793c3e5bccb0513"} Sep 14 09:45:23.440765 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.440736 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" event={"ID":"b3f931a3-7a96-44d7-beac-43811b45bb1a","Type":"ContainerStarted","Data":"11c8cce64c1b0a31bc3aaa260017fba9177d6fcf80a0d443c774e3a3e3f42f7c"} Sep 14 09:45:23.442849 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.442818 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnvrp" event={"ID":"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495","Type":"ContainerStarted","Data":"6ada0c91c065f181040092c25c48659c0de874c7e17d85e00f48249ca5804553"} Sep 14 09:45:23.442950 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.442864 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnvrp" event={"ID":"f4b05f60-7a5d-498c-9e7b-e0bc98d0d495","Type":"ContainerStarted","Data":"4ccdde52bbf0dcffd1973c84726057c1237ef15c07e06817cab085de0fdbea9f"} Sep 14 09:45:23.444738 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.444710 2550 generic.go:356] "Generic (PLEG): container finished" podID="f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36" containerID="26421646cb90414c3aa35cbca6c6199f535990cf79b02ca0ac53d106b0bf9b50" exitCode=0 Sep 14 09:45:23.444839 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.444758 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36","Type":"ContainerDied","Data":"26421646cb90414c3aa35cbca6c6199f535990cf79b02ca0ac53d106b0bf9b50"} Sep 14 09:45:23.444839 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.444814 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36","Type":"ContainerStarted","Data":"306da401827b0eeb3354a4601d865e48241a0011ab79a5b8917fb60b270923bf"} Sep 14 09:45:23.481402 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:23.481322 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lnvrp" podStartSLOduration=136.188446502 podStartE2EDuration="2m19.48130911s" totalImagesPullingTime="3.292862608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:23.480724532 +0000 UTC m=+140.175849839" watchObservedRunningTime="2026-09-14 09:45:23.48130911 +0000 UTC m=+140.176434413" Sep 14 09:45:28.483965 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.483933 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e","Type":"ContainerStarted","Data":"9dcd9c823d0d1555eeb47a3ad3bfa0239c0b7b76056a7f595ce7202916d88646"} Sep 14 09:45:28.484322 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.484018 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e","Type":"ContainerStarted","Data":"4ee5354be34ae19ace7e25c24b927dffa99d21e50555ed91e7829052156c2795"} Sep 14 09:45:28.489255 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.489211 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" event={"ID":"feb0e927-0450-41ab-9427-93dfe0392abe","Type":"ContainerStarted","Data":"9071a449807e5201b57f5480d844f89593e5dc2a17381b1555da0560197a80b5"} Sep 14 09:45:28.489361 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.489281 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" event={"ID":"feb0e927-0450-41ab-9427-93dfe0392abe","Type":"ContainerStarted","Data":"bd9e9551a29f50a4fbba827b10d6af231329d6df2bfeb25dd32cd2b046641eec"} Sep 14 09:45:28.489361 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.489320 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" event={"ID":"feb0e927-0450-41ab-9427-93dfe0392abe","Type":"ContainerStarted","Data":"d2d72efc7d291387a7c9f8be9baab4a4c95db3481c7bc5b2af3cc23a7683ab47"} Sep 14 09:45:28.489560 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.489527 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:28.494255 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.494095 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" event={"ID":"b3f931a3-7a96-44d7-beac-43811b45bb1a","Type":"ContainerStarted","Data":"788a58ecfbdc7283d1140b39955e16f2b2338e04140ffb564ca2ce751453de3f"} Sep 14 09:45:28.494255 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.494152 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" event={"ID":"b3f931a3-7a96-44d7-beac-43811b45bb1a","Type":"ContainerStarted","Data":"88ee429f3fabfb86721ba71b3337e55de11dd316df7333d7398fee995d2b4aa6"} Sep 14 09:45:28.494255 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.494181 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" event={"ID":"b3f931a3-7a96-44d7-beac-43811b45bb1a","Type":"ContainerStarted","Data":"1538013a8f7e5f3747bc83565a741d1e2268839aaa563d8d0ca1327685031f26"} Sep 14 09:45:28.497750 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.497672 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" Sep 14 09:45:28.502962 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.502814 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36","Type":"ContainerStarted","Data":"0ffe5af65584b1842a8c82dc7770c6995999191ceb12f2ed325243958bdf6e47"} Sep 14 09:45:28.523896 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.523842 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6cd55dd6f9-f5vrb" podStartSLOduration=3.403100467 podStartE2EDuration="11.523821209s" totalImagesPullingTime="8.120720742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:28.521381474 +0000 UTC m=+145.216506780" watchObservedRunningTime="2026-09-14 09:45:28.523821209 +0000 UTC m=+145.218946515" Sep 14 09:45:28.578026 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:28.576524 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5486df5f55-kj7bk" podStartSLOduration=3.646978137 podStartE2EDuration="8.576486683s" totalImagesPullingTime="4.929508546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:28.575353884 +0000 UTC m=+145.270479189" watchObservedRunningTime="2026-09-14 09:45:28.576486683 +0000 UTC m=+145.271611988" Sep 14 09:45:29.512265 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.512225 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36","Type":"ContainerStarted","Data":"218f2fe78e007069f8566dad38f4d28373ffccc7d3d2e3fcf220f137234b52df"} Sep 14 09:45:29.512901 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.512355 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36","Type":"ContainerStarted","Data":"4667e06be3f6f923feac637209c4ad526fb183b9592a7d3ccce97ef3fde0830b"} Sep 14 09:45:29.512901 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.512405 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36","Type":"ContainerStarted","Data":"f6827136d7c4315f546fbaef6469ae4fa6ad64341c17f7d194486ec485f7760d"} Sep 14 09:45:29.512901 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.512448 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36","Type":"ContainerStarted","Data":"0d60103dfc8747d5f08db04e6a1fdcc5bb9e118b67828791340fcf3e24f33ccd"} Sep 14 09:45:29.512901 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.512489 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36","Type":"ContainerStarted","Data":"216d371584aa85a862d93f637f7ae713cbf0f7fe9565223e1c84ca5fd990eac5"} Sep 14 09:45:29.515576 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.515550 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e","Type":"ContainerStarted","Data":"57e95eb47f2407284e0c84da36722e08a25c0649c0b20d098c8d5e338f26949c"} Sep 14 09:45:29.515705 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.515603 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e","Type":"ContainerStarted","Data":"ea3d5d1e227c53de7d62eefedb412050882c91b77e53cc7de96eaca5cabc4077"} Sep 14 09:45:29.515705 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.515630 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e","Type":"ContainerStarted","Data":"f78d3af13c85329afdcdb5da67e0d4e2033fb4ee2572e07d97e0c22b17a9565b"} Sep 14 09:45:29.515705 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.515670 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e632b34-a7bd-4f48-8750-bfdc1eca6b9e","Type":"ContainerStarted","Data":"3fdc42c289a1fc7b8bed155455f7e5136a047caa38a45b34f71d15c05693f87d"} Sep 14 09:45:29.601135 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.601062 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.859559551 podStartE2EDuration="13.601044835s" totalImagesPullingTime="7.731336671s" totalInitContainerRuntime="10.148613ms" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:29.599118959 +0000 UTC m=+146.294244274" watchObservedRunningTime="2026-09-14 09:45:29.601044835 +0000 UTC m=+146.296170141" Sep 14 09:45:29.601459 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:29.601427 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.749664049 podStartE2EDuration="7.60141693s" totalImagesPullingTime="4.829311409s" totalInitContainerRuntime="22.441472ms" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:29.563240665 +0000 UTC m=+146.258365969" watchObservedRunningTime="2026-09-14 09:45:29.60141693 +0000 UTC m=+146.296542234" Sep 14 09:45:32.485943 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:32.485900 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:45:33.060047 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.060014 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6956cc7fb6-pmxc7"] Sep 14 09:45:33.060283 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:45:33.060266 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" podUID="8b94aee3-35c3-45ee-aa73-dd0083f07b78" Sep 14 09:45:33.531160 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.530608 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:45:33.570995 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.570970 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:45:33.694523 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.694495 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-bound-sa-token\") pod \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " Sep 14 09:45:33.694672 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.694540 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-image-registry-private-configuration\") pod \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " Sep 14 09:45:33.694672 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.694584 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-trusted-ca\") pod \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " Sep 14 09:45:33.694672 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.694614 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-certificates\") pod \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " Sep 14 09:45:33.694672 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.694645 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-installation-pull-secrets\") pod \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " Sep 14 09:45:33.694885 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.694673 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b94aee3-35c3-45ee-aa73-dd0083f07b78-ca-trust-extracted\") pod \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " Sep 14 09:45:33.694885 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.694743 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9b6r\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-kube-api-access-n9b6r\") pod \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\" (UID: \"8b94aee3-35c3-45ee-aa73-dd0083f07b78\") " Sep 14 09:45:33.695020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.694991 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8b94aee3-35c3-45ee-aa73-dd0083f07b78" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 09:45:33.695069 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.694997 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8b94aee3-35c3-45ee-aa73-dd0083f07b78" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 09:45:33.695184 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.695161 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b94aee3-35c3-45ee-aa73-dd0083f07b78-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8b94aee3-35c3-45ee-aa73-dd0083f07b78" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 09:45:33.696940 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.696910 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8b94aee3-35c3-45ee-aa73-dd0083f07b78" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 09:45:33.697045 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.696973 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8b94aee3-35c3-45ee-aa73-dd0083f07b78" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 09:45:33.697166 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.697147 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-kube-api-access-n9b6r" (OuterVolumeSpecName: "kube-api-access-n9b6r") pod "8b94aee3-35c3-45ee-aa73-dd0083f07b78" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78"). InnerVolumeSpecName "kube-api-access-n9b6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 09:45:33.697235 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.697213 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8b94aee3-35c3-45ee-aa73-dd0083f07b78" (UID: "8b94aee3-35c3-45ee-aa73-dd0083f07b78"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 09:45:33.795900 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.795836 2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-certificates\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:33.795900 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.795860 2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-installation-pull-secrets\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:33.795900 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.795874 2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b94aee3-35c3-45ee-aa73-dd0083f07b78-ca-trust-extracted\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:33.795900 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.795889 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9b6r\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-kube-api-access-n9b6r\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:33.795900 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.795902 2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-bound-sa-token\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:33.796120 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.795916 2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b94aee3-35c3-45ee-aa73-dd0083f07b78-image-registry-private-configuration\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:33.796120 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:33.795929 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b94aee3-35c3-45ee-aa73-dd0083f07b78-trusted-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:34.534681 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:34.534628 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6956cc7fb6-pmxc7" Sep 14 09:45:34.568728 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:34.568701 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6956cc7fb6-pmxc7"] Sep 14 09:45:34.574346 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:34.574322 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6956cc7fb6-pmxc7"] Sep 14 09:45:34.702665 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:34.702625 2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b94aee3-35c3-45ee-aa73-dd0083f07b78-registry-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:35.823588 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:35.823552 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b94aee3-35c3-45ee-aa73-dd0083f07b78" path="/var/lib/kubelet/pods/8b94aee3-35c3-45ee-aa73-dd0083f07b78/volumes" Sep 14 09:45:40.189411 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:45:40.189373 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-q5n48" podUID="a7e4ee1c-60de-43ff-9b68-507c7ff58bdb" Sep 14 09:45:40.559010 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:40.558933 2550 generic.go:356] "Generic (PLEG): container finished" podID="5c6eb03b-bccc-48c8-81e5-94cae21943c6" containerID="4cb4164684de22b5886dc96c4809aae1479718d4555be25f387b3db895281da2" exitCode=0 Sep 14 09:45:40.559010 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:40.558992 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" event={"ID":"5c6eb03b-bccc-48c8-81e5-94cae21943c6","Type":"ContainerDied","Data":"4cb4164684de22b5886dc96c4809aae1479718d4555be25f387b3db895281da2"} Sep 14 09:45:40.559316 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:40.559303 2550 scope.go:117] "RemoveContainer" containerID="4cb4164684de22b5886dc96c4809aae1479718d4555be25f387b3db895281da2" Sep 14 09:45:41.566042 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:41.566006 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-wm6ht" event={"ID":"5c6eb03b-bccc-48c8-81e5-94cae21943c6","Type":"ContainerStarted","Data":"2f56b5f1dae1e3a0331051231392e33de31f5df64af5860400a6335b464daa1c"} Sep 14 09:45:45.088687 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.088619 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:45:45.091058 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.091026 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7e4ee1c-60de-43ff-9b68-507c7ff58bdb-metrics-tls\") pod \"dns-default-q5n48\" (UID: \"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb\") " pod="openshift-dns/dns-default-q5n48" Sep 14 09:45:45.496056 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.495997 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bdd99fc67-996zv" podUID="031f231a-7bda-4b8c-a595-3122a311443d" containerName="console" containerID="cri-o://921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39" gracePeriod=15 Sep 14 09:45:45.766755 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.766701 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:45.894928 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.894898 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-oauth-serving-cert\") pod \"031f231a-7bda-4b8c-a595-3122a311443d\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " Sep 14 09:45:45.895132 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.894937 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-serving-cert\") pod \"031f231a-7bda-4b8c-a595-3122a311443d\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " Sep 14 09:45:45.895132 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.895010 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-console-config\") pod \"031f231a-7bda-4b8c-a595-3122a311443d\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " Sep 14 09:45:45.895132 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.895059 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-oauth-config\") pod \"031f231a-7bda-4b8c-a595-3122a311443d\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " Sep 14 09:45:45.895132 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.895097 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2lsp\" (UniqueName: \"kubernetes.io/projected/031f231a-7bda-4b8c-a595-3122a311443d-kube-api-access-p2lsp\") pod \"031f231a-7bda-4b8c-a595-3122a311443d\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " Sep 14 09:45:45.895132 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.895130 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-service-ca\") pod \"031f231a-7bda-4b8c-a595-3122a311443d\" (UID: \"031f231a-7bda-4b8c-a595-3122a311443d\") " Sep 14 09:45:45.895397 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.895322 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "031f231a-7bda-4b8c-a595-3122a311443d" (UID: "031f231a-7bda-4b8c-a595-3122a311443d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 09:45:45.895484 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.895455 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-console-config" (OuterVolumeSpecName: "console-config") pod "031f231a-7bda-4b8c-a595-3122a311443d" (UID: "031f231a-7bda-4b8c-a595-3122a311443d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 09:45:45.895583 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.895556 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-service-ca" (OuterVolumeSpecName: "service-ca") pod "031f231a-7bda-4b8c-a595-3122a311443d" (UID: "031f231a-7bda-4b8c-a595-3122a311443d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 09:45:45.897280 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.897245 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "031f231a-7bda-4b8c-a595-3122a311443d" (UID: "031f231a-7bda-4b8c-a595-3122a311443d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 09:45:45.897367 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.897286 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "031f231a-7bda-4b8c-a595-3122a311443d" (UID: "031f231a-7bda-4b8c-a595-3122a311443d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 09:45:45.897367 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.897297 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031f231a-7bda-4b8c-a595-3122a311443d-kube-api-access-p2lsp" (OuterVolumeSpecName: "kube-api-access-p2lsp") pod "031f231a-7bda-4b8c-a595-3122a311443d" (UID: "031f231a-7bda-4b8c-a595-3122a311443d"). InnerVolumeSpecName "kube-api-access-p2lsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 09:45:45.996292 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.996266 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-oauth-serving-cert\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:45.996292 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.996288 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-serving-cert\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:45.996292 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.996297 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-console-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:45.996518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.996306 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/031f231a-7bda-4b8c-a595-3122a311443d-console-oauth-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:45.996518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.996314 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2lsp\" (UniqueName: \"kubernetes.io/projected/031f231a-7bda-4b8c-a595-3122a311443d-kube-api-access-p2lsp\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:45.996518 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:45.996323 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/031f231a-7bda-4b8c-a595-3122a311443d-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:45:46.586934 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:46.586899 2550 generic.go:356] "Generic (PLEG): container finished" podID="031f231a-7bda-4b8c-a595-3122a311443d" containerID="921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39" exitCode=0 Sep 14 09:45:46.587421 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:46.586940 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdd99fc67-996zv" Sep 14 09:45:46.587421 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:46.586986 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdd99fc67-996zv" event={"ID":"031f231a-7bda-4b8c-a595-3122a311443d","Type":"ContainerDied","Data":"921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39"} Sep 14 09:45:46.587421 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:46.587035 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdd99fc67-996zv" event={"ID":"031f231a-7bda-4b8c-a595-3122a311443d","Type":"ContainerDied","Data":"719d519c44228924eaa0a4e818fc625ccf94bf73c596eaf0e74587d973d84aa4"} Sep 14 09:45:46.587421 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:46.587057 2550 scope.go:117] "RemoveContainer" containerID="921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39" Sep 14 09:45:46.608476 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:46.608457 2550 scope.go:117] "RemoveContainer" containerID="921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39" Sep 14 09:45:46.608787 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:45:46.608763 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39\": container with ID starting with 921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39 not found: ID does not exist" containerID="921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39" Sep 14 09:45:46.608844 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:46.608795 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39"} err="failed to get container status \"921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39\": rpc error: code = NotFound desc = could not find container \"921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39\": container with ID starting with 921e63d436ee1ea346beec717257d38cbf1bebb03bdbf147c3291d41fa350b39 not found: ID does not exist" Sep 14 09:45:46.615330 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:46.615310 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdd99fc67-996zv"] Sep 14 09:45:46.619287 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:46.619266 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bdd99fc67-996zv"] Sep 14 09:45:47.825095 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:47.825060 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="031f231a-7bda-4b8c-a595-3122a311443d" path="/var/lib/kubelet/pods/031f231a-7bda-4b8c-a595-3122a311443d/volumes" Sep 14 09:45:54.818330 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:54.818294 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q5n48" Sep 14 09:45:54.821253 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:54.821235 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwxbr\"" Sep 14 09:45:54.829505 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:54.829490 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q5n48" Sep 14 09:45:55.135705 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:55.135682 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q5n48"] Sep 14 09:45:55.138128 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:45:55.138103 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e4ee1c_60de_43ff_9b68_507c7ff58bdb.slice/crio-e32418661aba4d87757b2b77e4903ccf23f0e7f3a95e379d9b5717612d035982 WatchSource:0}: Error finding container e32418661aba4d87757b2b77e4903ccf23f0e7f3a95e379d9b5717612d035982: Status 404 returned error can't find the container with id e32418661aba4d87757b2b77e4903ccf23f0e7f3a95e379d9b5717612d035982 Sep 14 09:45:55.624564 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:55.624532 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q5n48" event={"ID":"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb","Type":"ContainerStarted","Data":"e32418661aba4d87757b2b77e4903ccf23f0e7f3a95e379d9b5717612d035982"} Sep 14 09:45:57.637994 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:57.637911 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q5n48" event={"ID":"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb","Type":"ContainerStarted","Data":"2b8c7842795063722176df1e3ee91a96844017c6490b94d30feec8178945243b"} Sep 14 09:45:57.638384 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:57.637994 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q5n48" event={"ID":"a7e4ee1c-60de-43ff-9b68-507c7ff58bdb","Type":"ContainerStarted","Data":"404358ed82a4aea82ce04e17ddce5d7161e606dcfa44aba081ebc7430ec7e10e"} Sep 14 09:45:57.638384 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:57.638022 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q5n48" Sep 14 09:45:57.660210 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:45:57.660166 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q5n48" podStartSLOduration=138.594353933 podStartE2EDuration="2m20.660151724s" totalImagesPullingTime="2.065797791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:57.659499333 +0000 UTC m=+174.354624639" watchObservedRunningTime="2026-09-14 09:45:57.660151724 +0000 UTC m=+174.355277029" Sep 14 09:46:06.227998 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:46:06.227968 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q5n48_a7e4ee1c-60de-43ff-9b68-507c7ff58bdb/dns/0.log" Sep 14 09:46:06.234398 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:46:06.234365 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q5n48_a7e4ee1c-60de-43ff-9b68-507c7ff58bdb/kube-rbac-proxy/0.log" Sep 14 09:46:06.382612 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:46:06.382576 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4htsc_eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed/dns-node-resolver/0.log" Sep 14 09:46:07.645851 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:46:07.645823 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q5n48" Sep 14 09:46:22.485236 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:46:22.485188 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:46:22.504229 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:46:22.504204 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:46:22.759612 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:46:22.759541 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 09:48:03.738582 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:03.738553 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 09:48:03.739168 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:03.738829 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 09:48:03.755610 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:03.755588 2550 kubelet.go:1648] "Image garbage collection succeeded" Sep 14 09:48:58.163622 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.163545 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8"] Sep 14 09:48:58.164037 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.163860 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031f231a-7bda-4b8c-a595-3122a311443d" containerName="console" Sep 14 09:48:58.164037 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.163871 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="031f231a-7bda-4b8c-a595-3122a311443d" containerName="console" Sep 14 09:48:58.164037 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.163931 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="031f231a-7bda-4b8c-a595-3122a311443d" containerName="console" Sep 14 09:48:58.165749 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.165733 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" Sep 14 09:48:58.168203 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.168155 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5wt8l\"" Sep 14 09:48:58.168336 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.168215 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 14 09:48:58.169227 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.169211 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 14 09:48:58.177143 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.177126 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8"] Sep 14 09:48:58.329306 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.329273 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce610de1-c41f-4a1d-8783-ba4b5e39c628-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6pkc8\" (UID: \"ce610de1-c41f-4a1d-8783-ba4b5e39c628\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" Sep 14 09:48:58.329499 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.329393 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnjj\" (UniqueName: \"kubernetes.io/projected/ce610de1-c41f-4a1d-8783-ba4b5e39c628-kube-api-access-ttnjj\") pod \"openshift-lws-operator-fd8ccff4c-6pkc8\" (UID: \"ce610de1-c41f-4a1d-8783-ba4b5e39c628\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" Sep 14 09:48:58.430544 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.430471 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce610de1-c41f-4a1d-8783-ba4b5e39c628-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6pkc8\" (UID: \"ce610de1-c41f-4a1d-8783-ba4b5e39c628\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" Sep 14 09:48:58.430698 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.430554 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnjj\" (UniqueName: \"kubernetes.io/projected/ce610de1-c41f-4a1d-8783-ba4b5e39c628-kube-api-access-ttnjj\") pod \"openshift-lws-operator-fd8ccff4c-6pkc8\" (UID: \"ce610de1-c41f-4a1d-8783-ba4b5e39c628\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" Sep 14 09:48:58.430853 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.430833 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce610de1-c41f-4a1d-8783-ba4b5e39c628-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6pkc8\" (UID: \"ce610de1-c41f-4a1d-8783-ba4b5e39c628\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" Sep 14 09:48:58.440871 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.440846 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnjj\" (UniqueName: \"kubernetes.io/projected/ce610de1-c41f-4a1d-8783-ba4b5e39c628-kube-api-access-ttnjj\") pod \"openshift-lws-operator-fd8ccff4c-6pkc8\" (UID: \"ce610de1-c41f-4a1d-8783-ba4b5e39c628\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" Sep 14 09:48:58.483247 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.483225 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" Sep 14 09:48:58.757921 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.757892 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8"] Sep 14 09:48:58.759935 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:48:58.759905 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce610de1_c41f_4a1d_8783_ba4b5e39c628.slice/crio-d58c80c7c7e62ef365e9e9f1aa654ab28e7c8b8c204ab796cdc38b9e730b0391 WatchSource:0}: Error finding container d58c80c7c7e62ef365e9e9f1aa654ab28e7c8b8c204ab796cdc38b9e730b0391: Status 404 returned error can't find the container with id d58c80c7c7e62ef365e9e9f1aa654ab28e7c8b8c204ab796cdc38b9e730b0391 Sep 14 09:48:58.761383 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:58.761362 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 09:48:59.348850 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:48:59.348815 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" event={"ID":"ce610de1-c41f-4a1d-8783-ba4b5e39c628","Type":"ContainerStarted","Data":"d58c80c7c7e62ef365e9e9f1aa654ab28e7c8b8c204ab796cdc38b9e730b0391"} Sep 14 09:49:01.358804 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:01.358762 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" event={"ID":"ce610de1-c41f-4a1d-8783-ba4b5e39c628","Type":"ContainerStarted","Data":"246497cbabc7d6cb0fc475fdd70669d766efdcaca9ec4430e2a26d0b0cd11712"} Sep 14 09:49:01.375515 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:01.375464 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6pkc8" podStartSLOduration=0.98968034 podStartE2EDuration="3.375444081s" totalImagesPullingTime="2.385763741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:49:01.375170043 +0000 UTC m=+358.070295348" watchObservedRunningTime="2026-09-14 09:49:01.375444081 +0000 UTC m=+358.070569391" Sep 14 09:49:21.560567 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.560534 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7"] Sep 14 09:49:21.566938 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.566915 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.574602 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.574579 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 14 09:49:21.574753 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.574662 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 14 09:49:21.574753 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.574683 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n4vsj\"" Sep 14 09:49:21.575676 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.575660 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 14 09:49:21.576656 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.576630 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 14 09:49:21.608231 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.608206 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b42aeb4-6c44-485b-9129-8c64cad0a11a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.608305 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.608243 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b42aeb4-6c44-485b-9129-8c64cad0a11a-manifests\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.608305 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.608264 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b42aeb4-6c44-485b-9129-8c64cad0a11a-tmp\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.608374 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.608315 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b42aeb4-6c44-485b-9129-8c64cad0a11a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.608374 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.608337 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g8k2\" (UniqueName: \"kubernetes.io/projected/0b42aeb4-6c44-485b-9129-8c64cad0a11a-kube-api-access-9g8k2\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.618344 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.618319 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7"] Sep 14 09:49:21.709345 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.709315 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b42aeb4-6c44-485b-9129-8c64cad0a11a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.709491 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.709352 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g8k2\" (UniqueName: \"kubernetes.io/projected/0b42aeb4-6c44-485b-9129-8c64cad0a11a-kube-api-access-9g8k2\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.709491 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.709418 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b42aeb4-6c44-485b-9129-8c64cad0a11a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.709491 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.709458 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b42aeb4-6c44-485b-9129-8c64cad0a11a-manifests\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.709491 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.709487 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b42aeb4-6c44-485b-9129-8c64cad0a11a-tmp\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.709902 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.709880 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b42aeb4-6c44-485b-9129-8c64cad0a11a-manifests\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.711858 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.711839 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0b42aeb4-6c44-485b-9129-8c64cad0a11a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.712105 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.712088 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b42aeb4-6c44-485b-9129-8c64cad0a11a-tmp\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.712252 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.712237 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0b42aeb4-6c44-485b-9129-8c64cad0a11a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.720597 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.720576 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g8k2\" (UniqueName: \"kubernetes.io/projected/0b42aeb4-6c44-485b-9129-8c64cad0a11a-kube-api-access-9g8k2\") pod \"opendatahub-operator-controller-manager-84fdd9869c-kz6d7\" (UID: \"0b42aeb4-6c44-485b-9129-8c64cad0a11a\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:21.877336 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:21.877247 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:22.157733 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:22.157706 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7"] Sep 14 09:49:22.159533 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:49:22.159511 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b42aeb4_6c44_485b_9129_8c64cad0a11a.slice/crio-b9e409985c15f43149465bec3edd539b967d16e4a2ca7875a08f19833fa98f5c WatchSource:0}: Error finding container b9e409985c15f43149465bec3edd539b967d16e4a2ca7875a08f19833fa98f5c: Status 404 returned error can't find the container with id b9e409985c15f43149465bec3edd539b967d16e4a2ca7875a08f19833fa98f5c Sep 14 09:49:22.445400 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:22.445367 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" event={"ID":"0b42aeb4-6c44-485b-9129-8c64cad0a11a","Type":"ContainerStarted","Data":"b9e409985c15f43149465bec3edd539b967d16e4a2ca7875a08f19833fa98f5c"} Sep 14 09:49:25.465413 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:25.465371 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" event={"ID":"0b42aeb4-6c44-485b-9129-8c64cad0a11a","Type":"ContainerStarted","Data":"b0f2984d73a1c947d9beaeee04f91d2b27c2fec904cab1f3caeb8c469620a9ee"} Sep 14 09:49:26.471712 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:26.471609 2550 generic.go:356] "Generic (PLEG): container finished" podID="0b42aeb4-6c44-485b-9129-8c64cad0a11a" containerID="b0f2984d73a1c947d9beaeee04f91d2b27c2fec904cab1f3caeb8c469620a9ee" exitCode=0 Sep 14 09:49:26.471712 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:26.471697 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" event={"ID":"0b42aeb4-6c44-485b-9129-8c64cad0a11a","Type":"ContainerDied","Data":"b0f2984d73a1c947d9beaeee04f91d2b27c2fec904cab1f3caeb8c469620a9ee"} Sep 14 09:49:27.481216 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:27.481179 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" event={"ID":"0b42aeb4-6c44-485b-9129-8c64cad0a11a","Type":"ContainerStarted","Data":"b7f7a686004f526b5dfd75c3ce110fe5985bae6b3df33d50a6f78d207d6c5699"} Sep 14 09:49:27.481707 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:27.481253 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:27.506118 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:27.506075 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" podStartSLOduration=2.2127611209999998 podStartE2EDuration="6.506059728s" totalImagesPullingTime="3.253941716s" totalInitContainerRuntime="1.039356891s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:49:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:49:27.503727909 +0000 UTC m=+384.198853215" watchObservedRunningTime="2026-09-14 09:49:27.506059728 +0000 UTC m=+384.201185033" Sep 14 09:49:38.487473 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:38.487440 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-kz6d7" Sep 14 09:49:52.164274 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.164238 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l5m7r"] Sep 14 09:49:52.173466 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.173445 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:49:52.177276 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.177255 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ptxhs\"" Sep 14 09:49:52.177693 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.177677 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 14 09:49:52.183393 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.183373 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l5m7r"] Sep 14 09:49:52.273672 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.273615 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7w6f\" (UniqueName: \"kubernetes.io/projected/b2224a72-31fa-4624-ae8c-29fef205f2ee-kube-api-access-q7w6f\") pod \"odh-model-controller-7998c67dc4-l5m7r\" (UID: \"b2224a72-31fa-4624-ae8c-29fef205f2ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:49:52.273850 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.273794 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2224a72-31fa-4624-ae8c-29fef205f2ee-cert\") pod \"odh-model-controller-7998c67dc4-l5m7r\" (UID: \"b2224a72-31fa-4624-ae8c-29fef205f2ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:49:52.374971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.374934 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2224a72-31fa-4624-ae8c-29fef205f2ee-cert\") pod \"odh-model-controller-7998c67dc4-l5m7r\" (UID: \"b2224a72-31fa-4624-ae8c-29fef205f2ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:49:52.375165 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.374986 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7w6f\" (UniqueName: \"kubernetes.io/projected/b2224a72-31fa-4624-ae8c-29fef205f2ee-kube-api-access-q7w6f\") pod \"odh-model-controller-7998c67dc4-l5m7r\" (UID: \"b2224a72-31fa-4624-ae8c-29fef205f2ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:49:52.375165 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:49:52.375104 2550 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 14 09:49:52.375281 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:49:52.375234 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2224a72-31fa-4624-ae8c-29fef205f2ee-cert podName:b2224a72-31fa-4624-ae8c-29fef205f2ee nodeName:}" failed. No retries permitted until 2026-09-14 09:49:52.875210766 +0000 UTC m=+409.570336056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b2224a72-31fa-4624-ae8c-29fef205f2ee-cert") pod "odh-model-controller-7998c67dc4-l5m7r" (UID: "b2224a72-31fa-4624-ae8c-29fef205f2ee") : secret "odh-model-controller-webhook-cert" not found Sep 14 09:49:52.386659 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.386622 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7w6f\" (UniqueName: \"kubernetes.io/projected/b2224a72-31fa-4624-ae8c-29fef205f2ee-kube-api-access-q7w6f\") pod \"odh-model-controller-7998c67dc4-l5m7r\" (UID: \"b2224a72-31fa-4624-ae8c-29fef205f2ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:49:52.878826 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.878789 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2224a72-31fa-4624-ae8c-29fef205f2ee-cert\") pod \"odh-model-controller-7998c67dc4-l5m7r\" (UID: \"b2224a72-31fa-4624-ae8c-29fef205f2ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:49:52.881146 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:52.881128 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2224a72-31fa-4624-ae8c-29fef205f2ee-cert\") pod \"odh-model-controller-7998c67dc4-l5m7r\" (UID: \"b2224a72-31fa-4624-ae8c-29fef205f2ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:49:53.083534 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:53.083500 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:49:53.368169 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:53.368134 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l5m7r"] Sep 14 09:49:53.371554 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:49:53.371527 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2224a72_31fa_4624_ae8c_29fef205f2ee.slice/crio-41d8eb7a528a53474aa4017f8f54c1e70fde5ef44c4a0e1f9893f2487de15edf WatchSource:0}: Error finding container 41d8eb7a528a53474aa4017f8f54c1e70fde5ef44c4a0e1f9893f2487de15edf: Status 404 returned error can't find the container with id 41d8eb7a528a53474aa4017f8f54c1e70fde5ef44c4a0e1f9893f2487de15edf Sep 14 09:49:53.594910 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:53.594875 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" event={"ID":"b2224a72-31fa-4624-ae8c-29fef205f2ee","Type":"ContainerStarted","Data":"41d8eb7a528a53474aa4017f8f54c1e70fde5ef44c4a0e1f9893f2487de15edf"} Sep 14 09:49:56.612856 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:56.612771 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" event={"ID":"b2224a72-31fa-4624-ae8c-29fef205f2ee","Type":"ContainerStarted","Data":"de1a3796a18968a7077e4dbec7f318f03d8249cdd82eb8bb16de7596b1d87c5d"} Sep 14 09:49:56.613206 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:56.612897 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:49:56.632671 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:56.632605 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" podStartSLOduration=1.830554116 podStartE2EDuration="4.632587459s" totalImagesPullingTime="2.802033343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:49:56.631329318 +0000 UTC m=+413.326454627" watchObservedRunningTime="2026-09-14 09:49:56.632587459 +0000 UTC m=+413.327712793" Sep 14 09:49:59.489707 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.489664 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7"] Sep 14 09:49:59.493413 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.493390 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.498857 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.498834 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 14 09:49:59.499128 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.499111 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 14 09:49:59.499410 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.499395 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 14 09:49:59.499498 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.499428 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 14 09:49:59.499559 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.499506 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 14 09:49:59.499922 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.499904 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j8c6s\"" Sep 14 09:49:59.500027 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.499910 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 14 09:49:59.511413 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.511390 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7"] Sep 14 09:49:59.536902 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.536877 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78jql\" (UniqueName: \"kubernetes.io/projected/b568871b-3bca-40c4-8ca3-c90b8cb94373-kube-api-access-78jql\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.537027 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.536909 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.537027 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.536935 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.537140 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.537048 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b568871b-3bca-40c4-8ca3-c90b8cb94373-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.537140 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.537100 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b568871b-3bca-40c4-8ca3-c90b8cb94373-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.537227 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.537147 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.537227 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.537202 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.638547 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.638518 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.638547 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.638549 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78jql\" (UniqueName: \"kubernetes.io/projected/b568871b-3bca-40c4-8ca3-c90b8cb94373-kube-api-access-78jql\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.638788 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.638576 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.638788 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.638603 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.638788 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.638680 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b568871b-3bca-40c4-8ca3-c90b8cb94373-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.638788 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.638725 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b568871b-3bca-40c4-8ca3-c90b8cb94373-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.638788 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.638759 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.639410 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.639383 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.640945 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.640925 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.641030 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.640990 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b568871b-3bca-40c4-8ca3-c90b8cb94373-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.641030 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.641016 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b568871b-3bca-40c4-8ca3-c90b8cb94373-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.641102 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.640997 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.646436 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.646409 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78jql\" (UniqueName: \"kubernetes.io/projected/b568871b-3bca-40c4-8ca3-c90b8cb94373-kube-api-access-78jql\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.646674 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.646631 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b568871b-3bca-40c4-8ca3-c90b8cb94373-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-md4f7\" (UID: \"b568871b-3bca-40c4-8ca3-c90b8cb94373\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.655851 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.655828 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g"] Sep 14 09:49:59.659439 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.659421 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.662261 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.662238 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 14 09:49:59.662396 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.662249 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q9hzq\"" Sep 14 09:49:59.662396 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.662361 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 14 09:49:59.662503 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.662398 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 14 09:49:59.680859 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.680839 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g"] Sep 14 09:49:59.739969 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.739915 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e73301-ae3f-4061-8410-ec2bba9d5941-cert\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.739969 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.739946 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdtz\" (UniqueName: \"kubernetes.io/projected/a1e73301-ae3f-4061-8410-ec2bba9d5941-kube-api-access-7jdtz\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.740099 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.739987 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1e73301-ae3f-4061-8410-ec2bba9d5941-manager-config\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.740099 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.740044 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1e73301-ae3f-4061-8410-ec2bba9d5941-metrics-cert\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.803934 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.803909 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:49:59.841313 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.841278 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdtz\" (UniqueName: \"kubernetes.io/projected/a1e73301-ae3f-4061-8410-ec2bba9d5941-kube-api-access-7jdtz\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.841447 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.841365 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1e73301-ae3f-4061-8410-ec2bba9d5941-manager-config\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.841447 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.841410 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1e73301-ae3f-4061-8410-ec2bba9d5941-metrics-cert\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.841572 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.841464 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e73301-ae3f-4061-8410-ec2bba9d5941-cert\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.842196 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.842172 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1e73301-ae3f-4061-8410-ec2bba9d5941-manager-config\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.844728 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.844703 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1e73301-ae3f-4061-8410-ec2bba9d5941-metrics-cert\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.844816 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.844728 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e73301-ae3f-4061-8410-ec2bba9d5941-cert\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.860171 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.860150 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdtz\" (UniqueName: \"kubernetes.io/projected/a1e73301-ae3f-4061-8410-ec2bba9d5941-kube-api-access-7jdtz\") pod \"lws-controller-manager-5d6ddf947f-8bt8g\" (UID: \"a1e73301-ae3f-4061-8410-ec2bba9d5941\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:49:59.968924 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:49:59.968892 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:50:00.091437 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:00.091409 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7"] Sep 14 09:50:00.093943 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:50:00.093905 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb568871b_3bca_40c4_8ca3_c90b8cb94373.slice/crio-e5f6ddf6a215f3615943b07f39f14fc01efb52aa5c008bab4ac2732f75e5e213 WatchSource:0}: Error finding container e5f6ddf6a215f3615943b07f39f14fc01efb52aa5c008bab4ac2732f75e5e213: Status 404 returned error can't find the container with id e5f6ddf6a215f3615943b07f39f14fc01efb52aa5c008bab4ac2732f75e5e213 Sep 14 09:50:00.243409 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:00.243386 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g"] Sep 14 09:50:00.245198 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:50:00.245171 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e73301_ae3f_4061_8410_ec2bba9d5941.slice/crio-48d1db62777c6163d98818015dd28c0c52c4309f1e6bf02f3e57ba24c27040ee WatchSource:0}: Error finding container 48d1db62777c6163d98818015dd28c0c52c4309f1e6bf02f3e57ba24c27040ee: Status 404 returned error can't find the container with id 48d1db62777c6163d98818015dd28c0c52c4309f1e6bf02f3e57ba24c27040ee Sep 14 09:50:00.631682 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:00.631647 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" event={"ID":"a1e73301-ae3f-4061-8410-ec2bba9d5941","Type":"ContainerStarted","Data":"48d1db62777c6163d98818015dd28c0c52c4309f1e6bf02f3e57ba24c27040ee"} Sep 14 09:50:00.632878 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:00.632849 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" event={"ID":"b568871b-3bca-40c4-8ca3-c90b8cb94373","Type":"ContainerStarted","Data":"e5f6ddf6a215f3615943b07f39f14fc01efb52aa5c008bab4ac2732f75e5e213"} Sep 14 09:50:02.947828 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:02.947781 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 09:50:02.948107 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:02.947854 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 09:50:03.653154 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:03.653120 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" event={"ID":"b568871b-3bca-40c4-8ca3-c90b8cb94373","Type":"ContainerStarted","Data":"f070e2237c6baf8234218821fd4745a3a3b324cfef281d3c9c4e4a1574f0b411"} Sep 14 09:50:03.653352 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:03.653253 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:50:03.654586 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:03.654563 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" event={"ID":"a1e73301-ae3f-4061-8410-ec2bba9d5941","Type":"ContainerStarted","Data":"509ea2dc1f476621f1af4bff76e8a6b5fd97f7ad239b279ed396a2cef52c6e89"} Sep 14 09:50:03.654697 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:03.654687 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:50:03.681942 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:03.681903 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" podStartSLOduration=1.8303564319999999 podStartE2EDuration="4.68189069s" totalImagesPullingTime="2.851534258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:50:03.680124395 +0000 UTC m=+420.375249700" watchObservedRunningTime="2026-09-14 09:50:03.68189069 +0000 UTC m=+420.377015994" Sep 14 09:50:03.699510 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:03.699468 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" podStartSLOduration=1.997907669 podStartE2EDuration="4.699456093s" totalImagesPullingTime="2.701548424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:50:03.69753471 +0000 UTC m=+420.392660017" watchObservedRunningTime="2026-09-14 09:50:03.699456093 +0000 UTC m=+420.394581398" Sep 14 09:50:04.660692 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:04.660663 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md4f7" Sep 14 09:50:07.619922 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:07.619892 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:50:14.660956 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:50:14.660923 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-8bt8g" Sep 14 09:51:39.893208 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:39.893177 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l"] Sep 14 09:51:39.895600 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:39.895583 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:39.898487 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:39.898471 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 14 09:51:39.899069 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:39.899051 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zdclx\"" Sep 14 09:51:39.899109 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:39.899097 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 14 09:51:39.931673 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:39.931630 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l"] Sep 14 09:51:39.959355 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:39.959332 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l"] Sep 14 09:51:39.959610 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:51:39.959592 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-s58pr], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-s58pr]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" podUID="9a1e662a-0b2f-46ba-aa79-51f430a77845" Sep 14 09:51:39.972217 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:39.972193 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l"] Sep 14 09:51:39.993240 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:39.993205 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s58pr\" (UniqueName: \"kubernetes.io/projected/9a1e662a-0b2f-46ba-aa79-51f430a77845-kube-api-access-s58pr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9nm5l\" (UID: \"9a1e662a-0b2f-46ba-aa79-51f430a77845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:39.993380 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:39.993251 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a1e662a-0b2f-46ba-aa79-51f430a77845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9nm5l\" (UID: \"9a1e662a-0b2f-46ba-aa79-51f430a77845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:40.018422 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.018388 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs"] Sep 14 09:51:40.020976 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.020956 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:51:40.051469 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.051435 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs"] Sep 14 09:51:40.083930 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.083903 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:40.089996 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.089971 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:40.094001 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.093975 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s58pr\" (UniqueName: \"kubernetes.io/projected/9a1e662a-0b2f-46ba-aa79-51f430a77845-kube-api-access-s58pr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9nm5l\" (UID: \"9a1e662a-0b2f-46ba-aa79-51f430a77845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:40.094146 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.094008 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvbz\" (UniqueName: \"kubernetes.io/projected/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-kube-api-access-kzvbz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k95fs\" (UID: \"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:51:40.094146 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.094064 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a1e662a-0b2f-46ba-aa79-51f430a77845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9nm5l\" (UID: \"9a1e662a-0b2f-46ba-aa79-51f430a77845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:40.094271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.094224 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k95fs\" (UID: \"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:51:40.094449 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.094429 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a1e662a-0b2f-46ba-aa79-51f430a77845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9nm5l\" (UID: \"9a1e662a-0b2f-46ba-aa79-51f430a77845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:40.098145 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:51:40.098110 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-9nm5l\" is forbidden: User \"system:node:ip-10-0-135-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-40.ec2.internal' and this object" podUID="9a1e662a-0b2f-46ba-aa79-51f430a77845" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:40.108324 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:51:40.108304 2550 projected.go:196] Error preparing data for projected volume kube-api-access-s58pr for pod kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l: failed to fetch token: pod "kuadrant-operator-controller-manager-5769f99d9f-9nm5l" not found Sep 14 09:51:40.108425 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:51:40.108388 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a1e662a-0b2f-46ba-aa79-51f430a77845-kube-api-access-s58pr podName:9a1e662a-0b2f-46ba-aa79-51f430a77845 nodeName:}" failed. No retries permitted until 2026-09-14 09:51:40.608366263 +0000 UTC m=+517.303491548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s58pr" (UniqueName: "kubernetes.io/projected/9a1e662a-0b2f-46ba-aa79-51f430a77845-kube-api-access-s58pr") pod "kuadrant-operator-controller-manager-5769f99d9f-9nm5l" (UID: "9a1e662a-0b2f-46ba-aa79-51f430a77845") : failed to fetch token: pod "kuadrant-operator-controller-manager-5769f99d9f-9nm5l" not found Sep 14 09:51:40.194673 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.194621 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a1e662a-0b2f-46ba-aa79-51f430a77845-extensions-socket-volume\") pod \"9a1e662a-0b2f-46ba-aa79-51f430a77845\" (UID: \"9a1e662a-0b2f-46ba-aa79-51f430a77845\") " Sep 14 09:51:40.194845 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.194796 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvbz\" (UniqueName: \"kubernetes.io/projected/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-kube-api-access-kzvbz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k95fs\" (UID: \"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:51:40.194912 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.194871 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1e662a-0b2f-46ba-aa79-51f430a77845-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9a1e662a-0b2f-46ba-aa79-51f430a77845" (UID: "9a1e662a-0b2f-46ba-aa79-51f430a77845"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 09:51:40.194912 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.194886 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k95fs\" (UID: \"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:51:40.195014 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.194946 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s58pr\" (UniqueName: \"kubernetes.io/projected/9a1e662a-0b2f-46ba-aa79-51f430a77845-kube-api-access-s58pr\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:51:40.195014 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.194963 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a1e662a-0b2f-46ba-aa79-51f430a77845-extensions-socket-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:51:40.195265 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.195245 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k95fs\" (UID: \"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:51:40.210100 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.210081 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvbz\" (UniqueName: \"kubernetes.io/projected/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-kube-api-access-kzvbz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k95fs\" (UID: \"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:51:40.330852 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.330819 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:51:40.634087 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:51:40.634058 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8596cfd_3e6f_48e6_80be_e21ec8fc34d9.slice/crio-6f1e5bafc4dbf563f4608063cbe63abdcf3efbb83c48ad24985250f5e59c5679 WatchSource:0}: Error finding container 6f1e5bafc4dbf563f4608063cbe63abdcf3efbb83c48ad24985250f5e59c5679: Status 404 returned error can't find the container with id 6f1e5bafc4dbf563f4608063cbe63abdcf3efbb83c48ad24985250f5e59c5679 Sep 14 09:51:40.634208 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:40.634140 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs"] Sep 14 09:51:41.089698 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:41.089665 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" event={"ID":"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9","Type":"ContainerStarted","Data":"6f1e5bafc4dbf563f4608063cbe63abdcf3efbb83c48ad24985250f5e59c5679"} Sep 14 09:51:41.090067 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:41.089723 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:41.092948 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:51:41.092925 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-9nm5l\" is forbidden: User \"system:node:ip-10-0-135-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-40.ec2.internal' and this object" podUID="9a1e662a-0b2f-46ba-aa79-51f430a77845" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:41.099043 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:51:41.099020 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-9nm5l\" is forbidden: User \"system:node:ip-10-0-135-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-40.ec2.internal' and this object" podUID="9a1e662a-0b2f-46ba-aa79-51f430a77845" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9nm5l" Sep 14 09:51:41.826922 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:41.826836 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a1e662a-0b2f-46ba-aa79-51f430a77845" path="/var/lib/kubelet/pods/9a1e662a-0b2f-46ba-aa79-51f430a77845/volumes" Sep 14 09:51:45.114116 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:45.114032 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" event={"ID":"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9","Type":"ContainerStarted","Data":"a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4"} Sep 14 09:51:45.114508 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:45.114209 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:51:45.140385 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:45.140327 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" podStartSLOduration=1.996337182 podStartE2EDuration="6.140307195s" totalImagesPullingTime="4.143970013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:51:45.136963833 +0000 UTC m=+521.832089165" watchObservedRunningTime="2026-09-14 09:51:45.140307195 +0000 UTC m=+521.835432501" Sep 14 09:51:56.120911 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:51:56.120828 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:52:34.507608 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.507570 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-plj7l"] Sep 14 09:52:34.515428 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.512849 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" Sep 14 09:52:34.516354 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.516328 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 14 09:52:34.517197 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.517174 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 09:52:34.517305 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.517174 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 09:52:34.517305 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.517217 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-p6drb\"" Sep 14 09:52:34.526546 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.526525 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-plj7l"] Sep 14 09:52:34.587340 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.587309 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcmmq\" (UniqueName: \"kubernetes.io/projected/05274f1c-f432-455a-96fd-36f1f2bad255-kube-api-access-qcmmq\") pod \"postgres-69fb4594cf-plj7l\" (UID: \"05274f1c-f432-455a-96fd-36f1f2bad255\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" Sep 14 09:52:34.587483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.587365 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05274f1c-f432-455a-96fd-36f1f2bad255-data\") pod \"postgres-69fb4594cf-plj7l\" (UID: \"05274f1c-f432-455a-96fd-36f1f2bad255\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" Sep 14 09:52:34.688146 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.688101 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05274f1c-f432-455a-96fd-36f1f2bad255-data\") pod \"postgres-69fb4594cf-plj7l\" (UID: \"05274f1c-f432-455a-96fd-36f1f2bad255\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" Sep 14 09:52:34.688292 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.688192 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcmmq\" (UniqueName: \"kubernetes.io/projected/05274f1c-f432-455a-96fd-36f1f2bad255-kube-api-access-qcmmq\") pod \"postgres-69fb4594cf-plj7l\" (UID: \"05274f1c-f432-455a-96fd-36f1f2bad255\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" Sep 14 09:52:34.688470 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.688451 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05274f1c-f432-455a-96fd-36f1f2bad255-data\") pod \"postgres-69fb4594cf-plj7l\" (UID: \"05274f1c-f432-455a-96fd-36f1f2bad255\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" Sep 14 09:52:34.696696 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.696675 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcmmq\" (UniqueName: \"kubernetes.io/projected/05274f1c-f432-455a-96fd-36f1f2bad255-kube-api-access-qcmmq\") pod \"postgres-69fb4594cf-plj7l\" (UID: \"05274f1c-f432-455a-96fd-36f1f2bad255\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" Sep 14 09:52:34.826037 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:34.825974 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" Sep 14 09:52:35.095372 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:35.095275 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-plj7l"] Sep 14 09:52:35.098262 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:52:35.098230 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05274f1c_f432_455a_96fd_36f1f2bad255.slice/crio-9696038051cdb0e972187d1af546e9f40ae2ebf070dde37f81344bdc7f1f7b2d WatchSource:0}: Error finding container 9696038051cdb0e972187d1af546e9f40ae2ebf070dde37f81344bdc7f1f7b2d: Status 404 returned error can't find the container with id 9696038051cdb0e972187d1af546e9f40ae2ebf070dde37f81344bdc7f1f7b2d Sep 14 09:52:35.338274 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:35.338237 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" event={"ID":"05274f1c-f432-455a-96fd-36f1f2bad255","Type":"ContainerStarted","Data":"9696038051cdb0e972187d1af546e9f40ae2ebf070dde37f81344bdc7f1f7b2d"} Sep 14 09:52:40.368357 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:40.368325 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" event={"ID":"05274f1c-f432-455a-96fd-36f1f2bad255","Type":"ContainerStarted","Data":"be006896a81d800704be66c6ca2029b7e8cc8fb3144407d0a43fdc57e528b612"} Sep 14 09:52:40.368893 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:40.368503 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" Sep 14 09:52:40.387744 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:40.387699 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" podStartSLOduration=1.299656241 podStartE2EDuration="6.387685293s" totalImagesPullingTime="5.088029052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:52:40.386995523 +0000 UTC m=+577.082120828" watchObservedRunningTime="2026-09-14 09:52:40.387685293 +0000 UTC m=+577.082810628" Sep 14 09:52:46.401373 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:52:46.401343 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-plj7l" Sep 14 09:53:03.776558 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:03.776516 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 09:53:03.777072 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:03.777054 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 09:53:13.525461 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:13.525428 2550 generic.go:356] "Generic (PLEG): container finished" podID="b2224a72-31fa-4624-ae8c-29fef205f2ee" containerID="de1a3796a18968a7077e4dbec7f318f03d8249cdd82eb8bb16de7596b1d87c5d" exitCode=1 Sep 14 09:53:13.525918 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:13.525472 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" event={"ID":"b2224a72-31fa-4624-ae8c-29fef205f2ee","Type":"ContainerDied","Data":"de1a3796a18968a7077e4dbec7f318f03d8249cdd82eb8bb16de7596b1d87c5d"} Sep 14 09:53:13.525918 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:13.525797 2550 scope.go:117] "RemoveContainer" containerID="de1a3796a18968a7077e4dbec7f318f03d8249cdd82eb8bb16de7596b1d87c5d" Sep 14 09:53:14.532214 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:14.532177 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" event={"ID":"b2224a72-31fa-4624-ae8c-29fef205f2ee","Type":"ContainerStarted","Data":"1dceb33ad925fbe66d87b7c3edf41deb02020982cddf768fe29d3344887275e6"} Sep 14 09:53:14.532628 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:14.532397 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:53:14.533912 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:14.533891 2550 generic.go:356] "Generic (PLEG): container finished" podID="b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" containerID="a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4" exitCode=0 Sep 14 09:53:14.534011 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:14.533956 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" event={"ID":"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9","Type":"ContainerDied","Data":"a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4"} Sep 14 09:53:14.534228 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:14.534215 2550 scope.go:117] "RemoveContainer" containerID="a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4" Sep 14 09:53:15.540362 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:15.540329 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" event={"ID":"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9","Type":"ContainerStarted","Data":"05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c"} Sep 14 09:53:15.540756 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:15.540616 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:53:25.543190 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:25.543158 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-l5m7r" Sep 14 09:53:26.546743 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:26.546670 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 09:53:30.700093 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:30.700064 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j"] Sep 14 09:53:30.703527 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:30.703508 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" Sep 14 09:53:30.706067 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:30.706045 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 14 09:53:30.715187 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:30.715162 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j"] Sep 14 09:53:30.880173 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:30.880144 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e53b02d-3fed-459f-a481-c850ea69f9e1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dv48j\" (UID: \"6e53b02d-3fed-459f-a481-c850ea69f9e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" Sep 14 09:53:30.880353 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:30.880194 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gtl\" (UniqueName: \"kubernetes.io/projected/6e53b02d-3fed-459f-a481-c850ea69f9e1-kube-api-access-84gtl\") pod \"payload-pre-processing-59fbc47bf8-dv48j\" (UID: \"6e53b02d-3fed-459f-a481-c850ea69f9e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" Sep 14 09:53:30.981678 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:30.981572 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84gtl\" (UniqueName: \"kubernetes.io/projected/6e53b02d-3fed-459f-a481-c850ea69f9e1-kube-api-access-84gtl\") pod \"payload-pre-processing-59fbc47bf8-dv48j\" (UID: \"6e53b02d-3fed-459f-a481-c850ea69f9e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" Sep 14 09:53:30.981832 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:30.981744 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e53b02d-3fed-459f-a481-c850ea69f9e1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dv48j\" (UID: \"6e53b02d-3fed-459f-a481-c850ea69f9e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" Sep 14 09:53:30.982474 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:30.982451 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e53b02d-3fed-459f-a481-c850ea69f9e1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dv48j\" (UID: \"6e53b02d-3fed-459f-a481-c850ea69f9e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" Sep 14 09:53:30.989959 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:30.989924 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gtl\" (UniqueName: \"kubernetes.io/projected/6e53b02d-3fed-459f-a481-c850ea69f9e1-kube-api-access-84gtl\") pod \"payload-pre-processing-59fbc47bf8-dv48j\" (UID: \"6e53b02d-3fed-459f-a481-c850ea69f9e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" Sep 14 09:53:31.014843 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:31.014822 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" Sep 14 09:53:31.284250 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:31.284184 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j"] Sep 14 09:53:31.286661 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:53:31.286614 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e53b02d_3fed_459f_a481_c850ea69f9e1.slice/crio-5344a89b5958cdf38a66e0f2f9e16b289769b26cb59bf4f0679485dca3ecd899 WatchSource:0}: Error finding container 5344a89b5958cdf38a66e0f2f9e16b289769b26cb59bf4f0679485dca3ecd899: Status 404 returned error can't find the container with id 5344a89b5958cdf38a66e0f2f9e16b289769b26cb59bf4f0679485dca3ecd899 Sep 14 09:53:31.617434 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:31.617353 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" event={"ID":"6e53b02d-3fed-459f-a481-c850ea69f9e1","Type":"ContainerStarted","Data":"5344a89b5958cdf38a66e0f2f9e16b289769b26cb59bf4f0679485dca3ecd899"} Sep 14 09:53:33.633264 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:33.633231 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" event={"ID":"6e53b02d-3fed-459f-a481-c850ea69f9e1","Type":"ContainerStarted","Data":"d43aafacad418bb6982be58f8ef68a323bb92a9bb925ddb6b81ee39192b9acd0"} Sep 14 09:53:33.633674 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:33.633347 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" Sep 14 09:53:33.655566 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:33.655522 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" podStartSLOduration=1.431573855 podStartE2EDuration="3.655507235s" totalImagesPullingTime="2.22393338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:53:33.652772609 +0000 UTC m=+630.347897914" watchObservedRunningTime="2026-09-14 09:53:33.655507235 +0000 UTC m=+630.350632539" Sep 14 09:53:44.640616 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:53:44.640581 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dv48j" Sep 14 09:54:37.256778 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.256739 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g"] Sep 14 09:54:37.260967 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.260950 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.263820 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.263798 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 14 09:54:37.264927 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.264899 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8sh6p\"" Sep 14 09:54:37.264927 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.264899 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 14 09:54:37.265086 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.264946 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 14 09:54:37.272382 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.272357 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g"] Sep 14 09:54:37.357589 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.357556 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fgk\" (UniqueName: \"kubernetes.io/projected/9646de9f-9892-49b2-a276-930c859b9515-kube-api-access-h7fgk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.357589 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.357588 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9646de9f-9892-49b2-a276-930c859b9515-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.357804 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.357615 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.357804 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.357651 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.357804 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.357732 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.357804 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.357801 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.458587 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.458553 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.458749 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.458621 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.458749 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.458676 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fgk\" (UniqueName: \"kubernetes.io/projected/9646de9f-9892-49b2-a276-930c859b9515-kube-api-access-h7fgk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.458749 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.458702 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9646de9f-9892-49b2-a276-930c859b9515-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.458749 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.458740 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.458922 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.458768 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.459085 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.459055 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.459209 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.459071 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.459209 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.459152 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.460890 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.460867 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9646de9f-9892-49b2-a276-930c859b9515-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.461020 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.461004 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9646de9f-9892-49b2-a276-930c859b9515-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.469623 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.469602 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fgk\" (UniqueName: \"kubernetes.io/projected/9646de9f-9892-49b2-a276-930c859b9515-kube-api-access-h7fgk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ztn9g\" (UID: \"9646de9f-9892-49b2-a276-930c859b9515\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.573968 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.573904 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:37.848687 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.848663 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g"] Sep 14 09:54:37.850219 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:54:37.850192 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9646de9f_9892_49b2_a276_930c859b9515.slice/crio-f17cd09ab34d5fd25ba03fc1d4f44551e5993b5d8a0926c26f0b4c80187aeadb WatchSource:0}: Error finding container f17cd09ab34d5fd25ba03fc1d4f44551e5993b5d8a0926c26f0b4c80187aeadb: Status 404 returned error can't find the container with id f17cd09ab34d5fd25ba03fc1d4f44551e5993b5d8a0926c26f0b4c80187aeadb Sep 14 09:54:37.851548 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.851528 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 09:54:37.930824 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:37.930795 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" event={"ID":"9646de9f-9892-49b2-a276-930c859b9515","Type":"ContainerStarted","Data":"f17cd09ab34d5fd25ba03fc1d4f44551e5993b5d8a0926c26f0b4c80187aeadb"} Sep 14 09:54:38.939283 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:38.939251 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" event={"ID":"9646de9f-9892-49b2-a276-930c859b9515","Type":"ContainerStarted","Data":"f181491fb0ca4ec8a7451b5287139856258a69fc6a78055ba808f80031d5b06e"} Sep 14 09:54:38.963781 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:38.963732 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" podStartSLOduration=1.01123338 podStartE2EDuration="1.963715035s" totalImagesPullingTime="952.481655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:54:38.963051966 +0000 UTC m=+695.658177272" watchObservedRunningTime="2026-09-14 09:54:38.963715035 +0000 UTC m=+695.658840339" Sep 14 09:54:47.574158 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:47.574118 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:47.574609 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:47.574178 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:47.593038 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:47.592998 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:54:47.991866 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:54:47.991839 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ztn9g" Sep 14 09:57:26.071009 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.070974 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x"] Sep 14 09:57:26.075259 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.075222 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.079449 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.078990 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0e60f3-serving-cert\"" Sep 14 09:57:26.079449 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.078996 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 09:57:26.079449 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.078996 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-87zkx\"" Sep 14 09:57:26.087522 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.087501 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x"] Sep 14 09:57:26.138393 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.138365 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-metrics-certs\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.138561 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.138420 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.138561 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.138455 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d96dz\" (UniqueName: \"kubernetes.io/projected/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-kube-api-access-d96dz\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.138561 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.138549 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.239321 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.239286 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.239321 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.239321 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d96dz\" (UniqueName: \"kubernetes.io/projected/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-kube-api-access-d96dz\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.239510 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.239385 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.239510 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.239407 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-metrics-certs\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.240069 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.240045 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.241835 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.241814 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.241920 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.241844 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-metrics-certs\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.248713 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.248690 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d96dz\" (UniqueName: \"kubernetes.io/projected/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-kube-api-access-d96dz\") pod \"maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.389107 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.389023 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:26.689316 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.689279 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x"] Sep 14 09:57:26.695743 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:26.695715 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa3e4f7_68b3_46ae_b8b0_4f7fe79b96fe.slice/crio-cb05eb796f881e1cc7ed046f53326e5f068b8698feabe90daeede12ec778ae04 WatchSource:0}: Error finding container cb05eb796f881e1cc7ed046f53326e5f068b8698feabe90daeede12ec778ae04: Status 404 returned error can't find the container with id cb05eb796f881e1cc7ed046f53326e5f068b8698feabe90daeede12ec778ae04 Sep 14 09:57:26.739181 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:26.739144 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" event={"ID":"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe","Type":"ContainerStarted","Data":"cb05eb796f881e1cc7ed046f53326e5f068b8698feabe90daeede12ec778ae04"} Sep 14 09:57:27.107824 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.107750 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp"] Sep 14 09:57:27.112390 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.112373 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 09:57:27.115053 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.115028 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0e60f3-dockercfg-4fszf\"" Sep 14 09:57:27.115190 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.115060 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0e60f3\"" Sep 14 09:57:27.127152 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.127119 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp"] Sep 14 09:57:27.146922 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.146897 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrsh\" (UniqueName: \"kubernetes.io/projected/a9b07a80-34ab-4958-a61b-a94451edbb1b-kube-api-access-xgrsh\") pod \"payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp\" (UID: \"a9b07a80-34ab-4958-a61b-a94451edbb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 09:57:27.147025 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.147009 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b07a80-34ab-4958-a61b-a94451edbb1b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp\" (UID: \"a9b07a80-34ab-4958-a61b-a94451edbb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 09:57:27.159898 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.159876 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6"] Sep 14 09:57:27.163162 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.163146 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 09:57:27.175855 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.175829 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6"] Sep 14 09:57:27.248123 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.248078 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrsh\" (UniqueName: \"kubernetes.io/projected/a9b07a80-34ab-4958-a61b-a94451edbb1b-kube-api-access-xgrsh\") pod \"payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp\" (UID: \"a9b07a80-34ab-4958-a61b-a94451edbb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 09:57:27.248303 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.248275 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq98n\" (UniqueName: \"kubernetes.io/projected/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-kube-api-access-xq98n\") pod \"payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6\" (UID: \"b3e7cc8a-91aa-4dc7-a649-c329927f4f62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 09:57:27.248386 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.248323 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6\" (UID: \"b3e7cc8a-91aa-4dc7-a649-c329927f4f62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 09:57:27.248440 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.248381 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b07a80-34ab-4958-a61b-a94451edbb1b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp\" (UID: \"a9b07a80-34ab-4958-a61b-a94451edbb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 09:57:27.249214 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.249187 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b07a80-34ab-4958-a61b-a94451edbb1b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp\" (UID: \"a9b07a80-34ab-4958-a61b-a94451edbb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 09:57:27.258086 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.258057 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrsh\" (UniqueName: \"kubernetes.io/projected/a9b07a80-34ab-4958-a61b-a94451edbb1b-kube-api-access-xgrsh\") pod \"payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp\" (UID: \"a9b07a80-34ab-4958-a61b-a94451edbb1b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 09:57:27.349388 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.349346 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq98n\" (UniqueName: \"kubernetes.io/projected/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-kube-api-access-xq98n\") pod \"payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6\" (UID: \"b3e7cc8a-91aa-4dc7-a649-c329927f4f62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 09:57:27.349570 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.349405 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6\" (UID: \"b3e7cc8a-91aa-4dc7-a649-c329927f4f62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 09:57:27.350342 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.350312 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6\" (UID: \"b3e7cc8a-91aa-4dc7-a649-c329927f4f62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 09:57:27.359544 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.359476 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq98n\" (UniqueName: \"kubernetes.io/projected/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-kube-api-access-xq98n\") pod \"payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6\" (UID: \"b3e7cc8a-91aa-4dc7-a649-c329927f4f62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 09:57:27.424366 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.424141 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 09:57:27.475864 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.475834 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 09:57:27.732290 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.732213 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp"] Sep 14 09:57:27.734805 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:27.734764 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b07a80_34ab_4958_a61b_a94451edbb1b.slice/crio-fa758f208a96dbf0f55560d4d6ebfbc34188c0b065c01b8eee57bf06f2b2dbbc WatchSource:0}: Error finding container fa758f208a96dbf0f55560d4d6ebfbc34188c0b065c01b8eee57bf06f2b2dbbc: Status 404 returned error can't find the container with id fa758f208a96dbf0f55560d4d6ebfbc34188c0b065c01b8eee57bf06f2b2dbbc Sep 14 09:57:27.748378 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.748304 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" event={"ID":"a9b07a80-34ab-4958-a61b-a94451edbb1b","Type":"ContainerStarted","Data":"fa758f208a96dbf0f55560d4d6ebfbc34188c0b065c01b8eee57bf06f2b2dbbc"} Sep 14 09:57:27.798136 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:27.798108 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6"] Sep 14 09:57:27.801687 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:27.801655 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e7cc8a_91aa_4dc7_a649_c329927f4f62.slice/crio-4f82f7db3c6c9a9be3799f30c72f91cd45567de799faa59abf1b116d037a0f14 WatchSource:0}: Error finding container 4f82f7db3c6c9a9be3799f30c72f91cd45567de799faa59abf1b116d037a0f14: Status 404 returned error can't find the container with id 4f82f7db3c6c9a9be3799f30c72f91cd45567de799faa59abf1b116d037a0f14 Sep 14 09:57:28.341443 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.341409 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5"] Sep 14 09:57:28.351043 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.351008 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 09:57:28.355579 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.354963 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b6812e-dockercfg-cbqgz\"" Sep 14 09:57:28.355579 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.354975 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b6812e\"" Sep 14 09:57:28.357039 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.357013 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5"] Sep 14 09:57:28.418453 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.416616 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs"] Sep 14 09:57:28.422204 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.422181 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 09:57:28.437878 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.437700 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs"] Sep 14 09:57:28.461850 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.461819 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs\" (UID: \"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 09:57:28.462012 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.461856 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5529\" (UniqueName: \"kubernetes.io/projected/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-kube-api-access-r5529\") pod \"payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs\" (UID: \"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 09:57:28.462012 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.461937 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k27w\" (UniqueName: \"kubernetes.io/projected/2bf07770-24c1-4eaf-a696-de83cc22dcd2-kube-api-access-7k27w\") pod \"payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5\" (UID: \"2bf07770-24c1-4eaf-a696-de83cc22dcd2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 09:57:28.462012 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.461965 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf07770-24c1-4eaf-a696-de83cc22dcd2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5\" (UID: \"2bf07770-24c1-4eaf-a696-de83cc22dcd2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 09:57:28.563274 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.563219 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs\" (UID: \"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 09:57:28.563274 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.563269 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5529\" (UniqueName: \"kubernetes.io/projected/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-kube-api-access-r5529\") pod \"payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs\" (UID: \"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 09:57:28.563508 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.563352 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k27w\" (UniqueName: \"kubernetes.io/projected/2bf07770-24c1-4eaf-a696-de83cc22dcd2-kube-api-access-7k27w\") pod \"payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5\" (UID: \"2bf07770-24c1-4eaf-a696-de83cc22dcd2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 09:57:28.563508 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.563389 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf07770-24c1-4eaf-a696-de83cc22dcd2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5\" (UID: \"2bf07770-24c1-4eaf-a696-de83cc22dcd2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 09:57:28.564409 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.564268 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf07770-24c1-4eaf-a696-de83cc22dcd2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5\" (UID: \"2bf07770-24c1-4eaf-a696-de83cc22dcd2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 09:57:28.564865 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.564836 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs\" (UID: \"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 09:57:28.574796 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.574770 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5529\" (UniqueName: \"kubernetes.io/projected/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-kube-api-access-r5529\") pod \"payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs\" (UID: \"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 09:57:28.575380 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.575358 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k27w\" (UniqueName: \"kubernetes.io/projected/2bf07770-24c1-4eaf-a696-de83cc22dcd2-kube-api-access-7k27w\") pod \"payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5\" (UID: \"2bf07770-24c1-4eaf-a696-de83cc22dcd2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 09:57:28.666785 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.666757 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 09:57:28.736835 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.736798 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 09:57:28.755177 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.755142 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" event={"ID":"a9b07a80-34ab-4958-a61b-a94451edbb1b","Type":"ContainerStarted","Data":"858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40"} Sep 14 09:57:28.755336 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.755225 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 09:57:28.757203 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.757181 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" event={"ID":"b3e7cc8a-91aa-4dc7-a649-c329927f4f62","Type":"ContainerStarted","Data":"7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772"} Sep 14 09:57:28.757305 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.757217 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" event={"ID":"b3e7cc8a-91aa-4dc7-a649-c329927f4f62","Type":"ContainerStarted","Data":"4f82f7db3c6c9a9be3799f30c72f91cd45567de799faa59abf1b116d037a0f14"} Sep 14 09:57:28.757364 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.757319 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 09:57:28.785203 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.782801 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" podStartSLOduration=1.7827810450000001 podStartE2EDuration="1.782781045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:28.781377817 +0000 UTC m=+865.476503125" watchObservedRunningTime="2026-09-14 09:57:28.782781045 +0000 UTC m=+865.477906353" Sep 14 09:57:28.815920 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:28.815861 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" podStartSLOduration=1.815837405 podStartE2EDuration="1.815837405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:28.813347747 +0000 UTC m=+865.508473053" watchObservedRunningTime="2026-09-14 09:57:28.815837405 +0000 UTC m=+865.510962711" Sep 14 09:57:29.562438 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.562403 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5"] Sep 14 09:57:29.563780 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:29.563736 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf07770_24c1_4eaf_a696_de83cc22dcd2.slice/crio-46bbd5b525cc701643e728ecd466c15c865f570714784bacd8383972122b3b18 WatchSource:0}: Error finding container 46bbd5b525cc701643e728ecd466c15c865f570714784bacd8383972122b3b18: Status 404 returned error can't find the container with id 46bbd5b525cc701643e728ecd466c15c865f570714784bacd8383972122b3b18 Sep 14 09:57:29.580100 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.580070 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs"] Sep 14 09:57:29.581159 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:29.581137 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6b37aa_2afd_49aa_aa1c_0d33d78abb88.slice/crio-a96fa6435f8a4635da46a15234e91616b75b18a29cd9c8515eefb88275c971d3 WatchSource:0}: Error finding container a96fa6435f8a4635da46a15234e91616b75b18a29cd9c8515eefb88275c971d3: Status 404 returned error can't find the container with id a96fa6435f8a4635da46a15234e91616b75b18a29cd9c8515eefb88275c971d3 Sep 14 09:57:29.766710 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.766668 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" event={"ID":"2bf07770-24c1-4eaf-a696-de83cc22dcd2","Type":"ContainerStarted","Data":"d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31"} Sep 14 09:57:29.766901 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.766739 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 09:57:29.766901 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.766765 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" event={"ID":"2bf07770-24c1-4eaf-a696-de83cc22dcd2","Type":"ContainerStarted","Data":"46bbd5b525cc701643e728ecd466c15c865f570714784bacd8383972122b3b18"} Sep 14 09:57:29.768550 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.768433 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" event={"ID":"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88","Type":"ContainerStarted","Data":"c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3"} Sep 14 09:57:29.768550 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.768489 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" event={"ID":"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88","Type":"ContainerStarted","Data":"a96fa6435f8a4635da46a15234e91616b75b18a29cd9c8515eefb88275c971d3"} Sep 14 09:57:29.768550 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.768517 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 09:57:29.770143 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.770120 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" event={"ID":"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe","Type":"ContainerStarted","Data":"b736b40fdd723e0400e2e29164f0631d51b980fa3bdd8cbb0b780b36689c5b57"} Sep 14 09:57:29.814604 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.814536 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr"] Sep 14 09:57:29.819068 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.818482 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 09:57:29.823514 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.823489 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2901ec\"" Sep 14 09:57:29.823751 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.823732 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2901ec-dockercfg-g9m7m\"" Sep 14 09:57:29.827053 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.826627 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" podStartSLOduration=1.826610809 podStartE2EDuration="1.826610809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:29.824202879 +0000 UTC m=+866.519328185" watchObservedRunningTime="2026-09-14 09:57:29.826610809 +0000 UTC m=+866.521736115" Sep 14 09:57:29.852748 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.852718 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr"] Sep 14 09:57:29.875393 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.875367 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5dba100-838a-49dc-bdee-9a0883e98396-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr\" (UID: \"b5dba100-838a-49dc-bdee-9a0883e98396\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 09:57:29.875683 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.875659 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44djp\" (UniqueName: \"kubernetes.io/projected/b5dba100-838a-49dc-bdee-9a0883e98396-kube-api-access-44djp\") pod \"payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr\" (UID: \"b5dba100-838a-49dc-bdee-9a0883e98396\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 09:57:29.890646 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.890584 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" podStartSLOduration=1.2772048599999999 podStartE2EDuration="3.890569755s" totalImagesPullingTime="2.613364895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:29.883586118 +0000 UTC m=+866.578711423" watchObservedRunningTime="2026-09-14 09:57:29.890569755 +0000 UTC m=+866.585695060" Sep 14 09:57:29.926512 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.926482 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k"] Sep 14 09:57:29.930325 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.930305 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 09:57:29.935149 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.935094 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" podStartSLOduration=1.935077411 podStartE2EDuration="1.935077411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:29.933198967 +0000 UTC m=+866.628324273" watchObservedRunningTime="2026-09-14 09:57:29.935077411 +0000 UTC m=+866.630202719" Sep 14 09:57:29.951436 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.951411 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k"] Sep 14 09:57:29.977191 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.977162 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc763f21-0c0e-4ae8-bc3b-ef835738f829-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k\" (UID: \"fc763f21-0c0e-4ae8-bc3b-ef835738f829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 09:57:29.977334 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.977209 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5dba100-838a-49dc-bdee-9a0883e98396-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr\" (UID: \"b5dba100-838a-49dc-bdee-9a0883e98396\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 09:57:29.977334 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.977279 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44djp\" (UniqueName: \"kubernetes.io/projected/b5dba100-838a-49dc-bdee-9a0883e98396-kube-api-access-44djp\") pod \"payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr\" (UID: \"b5dba100-838a-49dc-bdee-9a0883e98396\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 09:57:29.977334 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.977329 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sd9\" (UniqueName: \"kubernetes.io/projected/fc763f21-0c0e-4ae8-bc3b-ef835738f829-kube-api-access-f8sd9\") pod \"payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k\" (UID: \"fc763f21-0c0e-4ae8-bc3b-ef835738f829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 09:57:29.977903 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:29.977885 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5dba100-838a-49dc-bdee-9a0883e98396-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr\" (UID: \"b5dba100-838a-49dc-bdee-9a0883e98396\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 09:57:30.061003 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.060965 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44djp\" (UniqueName: \"kubernetes.io/projected/b5dba100-838a-49dc-bdee-9a0883e98396-kube-api-access-44djp\") pod \"payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr\" (UID: \"b5dba100-838a-49dc-bdee-9a0883e98396\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 09:57:30.078725 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.078626 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc763f21-0c0e-4ae8-bc3b-ef835738f829-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k\" (UID: \"fc763f21-0c0e-4ae8-bc3b-ef835738f829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 09:57:30.078876 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.078774 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sd9\" (UniqueName: \"kubernetes.io/projected/fc763f21-0c0e-4ae8-bc3b-ef835738f829-kube-api-access-f8sd9\") pod \"payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k\" (UID: \"fc763f21-0c0e-4ae8-bc3b-ef835738f829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 09:57:30.079343 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.079323 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc763f21-0c0e-4ae8-bc3b-ef835738f829-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k\" (UID: \"fc763f21-0c0e-4ae8-bc3b-ef835738f829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 09:57:30.116263 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.116218 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sd9\" (UniqueName: \"kubernetes.io/projected/fc763f21-0c0e-4ae8-bc3b-ef835738f829-kube-api-access-f8sd9\") pod \"payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k\" (UID: \"fc763f21-0c0e-4ae8-bc3b-ef835738f829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 09:57:30.135988 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.135953 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 09:57:30.241701 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.241669 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 09:57:30.429880 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.429409 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr"] Sep 14 09:57:30.433993 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:30.433769 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5dba100_838a_49dc_bdee_9a0883e98396.slice/crio-5560d6c3a02813dfa619b1d9d3c997c415a34c395ffc6d4bb76643c814512201 WatchSource:0}: Error finding container 5560d6c3a02813dfa619b1d9d3c997c415a34c395ffc6d4bb76643c814512201: Status 404 returned error can't find the container with id 5560d6c3a02813dfa619b1d9d3c997c415a34c395ffc6d4bb76643c814512201 Sep 14 09:57:30.527483 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.527378 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k"] Sep 14 09:57:30.530835 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:30.530803 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc763f21_0c0e_4ae8_bc3b_ef835738f829.slice/crio-9911e6140c56c1d257e46cf691d33e5947b9357674964a354226166bc6e9e851 WatchSource:0}: Error finding container 9911e6140c56c1d257e46cf691d33e5947b9357674964a354226166bc6e9e851: Status 404 returned error can't find the container with id 9911e6140c56c1d257e46cf691d33e5947b9357674964a354226166bc6e9e851 Sep 14 09:57:30.779624 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.779530 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" event={"ID":"b5dba100-838a-49dc-bdee-9a0883e98396","Type":"ContainerStarted","Data":"45db35ea48a476f2260fa83b751ab86e3a257e8b3e0dda451f67666fbda37d3e"} Sep 14 09:57:30.779624 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.779591 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" event={"ID":"b5dba100-838a-49dc-bdee-9a0883e98396","Type":"ContainerStarted","Data":"5560d6c3a02813dfa619b1d9d3c997c415a34c395ffc6d4bb76643c814512201"} Sep 14 09:57:30.779624 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.779625 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 09:57:30.781259 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.781227 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" event={"ID":"fc763f21-0c0e-4ae8-bc3b-ef835738f829","Type":"ContainerStarted","Data":"7a4b802fdbcd6f7f0d62df4e189f5123b9cef7c2086a9ba373c63dcd8483e7f3"} Sep 14 09:57:30.781381 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.781285 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" event={"ID":"fc763f21-0c0e-4ae8-bc3b-ef835738f829","Type":"ContainerStarted","Data":"9911e6140c56c1d257e46cf691d33e5947b9357674964a354226166bc6e9e851"} Sep 14 09:57:30.781615 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.781588 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 09:57:30.781758 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.781662 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:30.808971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.808928 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" podStartSLOduration=1.808914051 podStartE2EDuration="1.808914051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:30.807133905 +0000 UTC m=+867.502259193" watchObservedRunningTime="2026-09-14 09:57:30.808914051 +0000 UTC m=+867.504039356" Sep 14 09:57:30.836099 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:30.836024 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" podStartSLOduration=1.8360042970000001 podStartE2EDuration="1.836004297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:30.83408467 +0000 UTC m=+867.529209981" watchObservedRunningTime="2026-09-14 09:57:30.836004297 +0000 UTC m=+867.531129609" Sep 14 09:57:36.793791 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:36.793759 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 09:57:39.241793 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.241754 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg"] Sep 14 09:57:39.245459 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.245437 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 09:57:39.248468 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.248436 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-8c98aa\"" Sep 14 09:57:39.248610 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.248497 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-8c98aa-dockercfg-72ssd\"" Sep 14 09:57:39.260714 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.260689 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg"] Sep 14 09:57:39.367567 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.367532 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zhn\" (UniqueName: \"kubernetes.io/projected/7879e468-9b31-49dc-a21a-e5850db2365d-kube-api-access-v4zhn\") pod \"payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg\" (UID: \"7879e468-9b31-49dc-a21a-e5850db2365d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 09:57:39.367846 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.367590 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7879e468-9b31-49dc-a21a-e5850db2365d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg\" (UID: \"7879e468-9b31-49dc-a21a-e5850db2365d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 09:57:39.468765 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.468724 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zhn\" (UniqueName: \"kubernetes.io/projected/7879e468-9b31-49dc-a21a-e5850db2365d-kube-api-access-v4zhn\") pod \"payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg\" (UID: \"7879e468-9b31-49dc-a21a-e5850db2365d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 09:57:39.468939 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.468777 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7879e468-9b31-49dc-a21a-e5850db2365d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg\" (UID: \"7879e468-9b31-49dc-a21a-e5850db2365d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 09:57:39.469515 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.469492 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7879e468-9b31-49dc-a21a-e5850db2365d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg\" (UID: \"7879e468-9b31-49dc-a21a-e5850db2365d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 09:57:39.479200 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.479174 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zhn\" (UniqueName: \"kubernetes.io/projected/7879e468-9b31-49dc-a21a-e5850db2365d-kube-api-access-v4zhn\") pod \"payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg\" (UID: \"7879e468-9b31-49dc-a21a-e5850db2365d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 09:57:39.555777 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.555683 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 09:57:39.772202 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.772173 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 09:57:39.772342 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.772250 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 09:57:39.861068 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:39.860985 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg"] Sep 14 09:57:39.863677 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:39.863627 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7879e468_9b31_49dc_a21a_e5850db2365d.slice/crio-6044b75545e5c100a7cc712ae997d03eb8d6507b5ef60b0bfd0e4ceccaf9f5f5 WatchSource:0}: Error finding container 6044b75545e5c100a7cc712ae997d03eb8d6507b5ef60b0bfd0e4ceccaf9f5f5: Status 404 returned error can't find the container with id 6044b75545e5c100a7cc712ae997d03eb8d6507b5ef60b0bfd0e4ceccaf9f5f5 Sep 14 09:57:40.783207 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.783181 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 09:57:40.783594 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.783249 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 09:57:40.850005 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.849966 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" event={"ID":"7879e468-9b31-49dc-a21a-e5850db2365d","Type":"ContainerStarted","Data":"fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb"} Sep 14 09:57:40.850192 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.850036 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" event={"ID":"7879e468-9b31-49dc-a21a-e5850db2365d","Type":"ContainerStarted","Data":"6044b75545e5c100a7cc712ae997d03eb8d6507b5ef60b0bfd0e4ceccaf9f5f5"} Sep 14 09:57:40.850192 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.850167 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 09:57:40.873306 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.873251 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" podStartSLOduration=1.873232096 podStartE2EDuration="1.873232096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:40.87013418 +0000 UTC m=+877.565259485" watchObservedRunningTime="2026-09-14 09:57:40.873232096 +0000 UTC m=+877.568357403" Sep 14 09:57:40.917613 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.917579 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5"] Sep 14 09:57:40.921488 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.921468 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:57:40.924568 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.924551 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9eed367b-dockercfg-gfrds\"" Sep 14 09:57:40.924886 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.924870 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9eed367b\"" Sep 14 09:57:40.935255 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.935230 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg"] Sep 14 09:57:40.940171 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.940149 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5"] Sep 14 09:57:40.940271 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.940240 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:57:40.955865 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.955843 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg"] Sep 14 09:57:40.984268 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.984231 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trldz\" (UniqueName: \"kubernetes.io/projected/42379a1a-bbca-410a-8b56-efb32fffd6ad-kube-api-access-trldz\") pod \"payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5\" (UID: \"42379a1a-bbca-410a-8b56-efb32fffd6ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:57:40.984843 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:40.984533 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42379a1a-bbca-410a-8b56-efb32fffd6ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5\" (UID: \"42379a1a-bbca-410a-8b56-efb32fffd6ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:57:41.085486 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.085407 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29de6996-dcd5-4718-a4cf-c52d7c78e11f-plugins-config-volume\") pod \"payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg\" (UID: \"29de6996-dcd5-4718-a4cf-c52d7c78e11f\") " pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:57:41.085486 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.085446 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trldz\" (UniqueName: \"kubernetes.io/projected/42379a1a-bbca-410a-8b56-efb32fffd6ad-kube-api-access-trldz\") pod \"payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5\" (UID: \"42379a1a-bbca-410a-8b56-efb32fffd6ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:57:41.085486 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.085471 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42379a1a-bbca-410a-8b56-efb32fffd6ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5\" (UID: \"42379a1a-bbca-410a-8b56-efb32fffd6ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:57:41.085776 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.085603 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsjqw\" (UniqueName: \"kubernetes.io/projected/29de6996-dcd5-4718-a4cf-c52d7c78e11f-kube-api-access-nsjqw\") pod \"payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg\" (UID: \"29de6996-dcd5-4718-a4cf-c52d7c78e11f\") " pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:57:41.086193 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.086173 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42379a1a-bbca-410a-8b56-efb32fffd6ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5\" (UID: \"42379a1a-bbca-410a-8b56-efb32fffd6ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:57:41.095432 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.095412 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trldz\" (UniqueName: \"kubernetes.io/projected/42379a1a-bbca-410a-8b56-efb32fffd6ad-kube-api-access-trldz\") pod \"payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5\" (UID: \"42379a1a-bbca-410a-8b56-efb32fffd6ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:57:41.186520 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.186484 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29de6996-dcd5-4718-a4cf-c52d7c78e11f-plugins-config-volume\") pod \"payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg\" (UID: \"29de6996-dcd5-4718-a4cf-c52d7c78e11f\") " pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:57:41.186687 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.186549 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsjqw\" (UniqueName: \"kubernetes.io/projected/29de6996-dcd5-4718-a4cf-c52d7c78e11f-kube-api-access-nsjqw\") pod \"payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg\" (UID: \"29de6996-dcd5-4718-a4cf-c52d7c78e11f\") " pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:57:41.187209 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.187188 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29de6996-dcd5-4718-a4cf-c52d7c78e11f-plugins-config-volume\") pod \"payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg\" (UID: \"29de6996-dcd5-4718-a4cf-c52d7c78e11f\") " pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:57:41.198414 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.198393 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsjqw\" (UniqueName: \"kubernetes.io/projected/29de6996-dcd5-4718-a4cf-c52d7c78e11f-kube-api-access-nsjqw\") pod \"payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg\" (UID: \"29de6996-dcd5-4718-a4cf-c52d7c78e11f\") " pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:57:41.232687 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.232662 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:57:41.251331 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.251266 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:57:41.509147 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.509111 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5"] Sep 14 09:57:41.513497 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:41.513464 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42379a1a_bbca_410a_8b56_efb32fffd6ad.slice/crio-2f7afc72b2322ac4adcb8614b56cd51124ec9371aafe482ccccad958fcda59c5 WatchSource:0}: Error finding container 2f7afc72b2322ac4adcb8614b56cd51124ec9371aafe482ccccad958fcda59c5: Status 404 returned error can't find the container with id 2f7afc72b2322ac4adcb8614b56cd51124ec9371aafe482ccccad958fcda59c5 Sep 14 09:57:41.537335 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.537309 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg"] Sep 14 09:57:41.539093 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:41.539060 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29de6996_dcd5_4718_a4cf_c52d7c78e11f.slice/crio-72c63b470b4577743baf7e82297f1ba0fecbb4be95e7f2a15d93f9b1922ecdf3 WatchSource:0}: Error finding container 72c63b470b4577743baf7e82297f1ba0fecbb4be95e7f2a15d93f9b1922ecdf3: Status 404 returned error can't find the container with id 72c63b470b4577743baf7e82297f1ba0fecbb4be95e7f2a15d93f9b1922ecdf3 Sep 14 09:57:41.790302 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.790224 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 09:57:41.859772 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.859740 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" event={"ID":"42379a1a-bbca-410a-8b56-efb32fffd6ad","Type":"ContainerStarted","Data":"c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2"} Sep 14 09:57:41.859996 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.859980 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" event={"ID":"42379a1a-bbca-410a-8b56-efb32fffd6ad","Type":"ContainerStarted","Data":"2f7afc72b2322ac4adcb8614b56cd51124ec9371aafe482ccccad958fcda59c5"} Sep 14 09:57:41.860279 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.860264 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:57:41.862538 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.862516 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" event={"ID":"29de6996-dcd5-4718-a4cf-c52d7c78e11f","Type":"ContainerStarted","Data":"b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3"} Sep 14 09:57:41.862835 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.862813 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:57:41.863040 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.863004 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" event={"ID":"29de6996-dcd5-4718-a4cf-c52d7c78e11f","Type":"ContainerStarted","Data":"72c63b470b4577743baf7e82297f1ba0fecbb4be95e7f2a15d93f9b1922ecdf3"} Sep 14 09:57:41.897011 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.896962 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" podStartSLOduration=1.8969445729999999 podStartE2EDuration="1.896944573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:41.895151089 +0000 UTC m=+878.590276394" watchObservedRunningTime="2026-09-14 09:57:41.896944573 +0000 UTC m=+878.592069878" Sep 14 09:57:41.933508 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:41.933443 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" podStartSLOduration=1.933421646 podStartE2EDuration="1.933421646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:41.930439217 +0000 UTC m=+878.625564525" watchObservedRunningTime="2026-09-14 09:57:41.933421646 +0000 UTC m=+878.628546954" Sep 14 09:57:42.795239 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:42.795205 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 09:57:51.864694 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:51.864659 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 09:57:53.875302 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:53.875273 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:57:53.875664 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:53.875339 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:57:53.960173 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:53.960143 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2"] Sep 14 09:57:53.969108 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:53.969072 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:53.973105 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:53.973084 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-24873d-serving-cert\"" Sep 14 09:57:53.979381 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:53.979353 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2"] Sep 14 09:57:54.110606 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.110563 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-maas-api-tls\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.110795 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.110612 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb2dcff-114f-48d9-9baf-401b6ac59183-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.110795 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.110766 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk2gs\" (UniqueName: \"kubernetes.io/projected/afb2dcff-114f-48d9-9baf-401b6ac59183-kube-api-access-fk2gs\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.110912 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.110821 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-metrics-certs\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.212057 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.212020 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk2gs\" (UniqueName: \"kubernetes.io/projected/afb2dcff-114f-48d9-9baf-401b6ac59183-kube-api-access-fk2gs\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.212194 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.212071 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-metrics-certs\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.212194 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.212109 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-maas-api-tls\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.212194 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.212132 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb2dcff-114f-48d9-9baf-401b6ac59183-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.212978 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.212953 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb2dcff-114f-48d9-9baf-401b6ac59183-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.214527 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.214506 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-maas-api-tls\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.214702 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.214674 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-metrics-certs\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.220953 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.220923 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk2gs\" (UniqueName: \"kubernetes.io/projected/afb2dcff-114f-48d9-9baf-401b6ac59183-kube-api-access-fk2gs\") pod \"maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.312071 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.312040 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:54.589526 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.589501 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2"] Sep 14 09:57:54.590668 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:54.590629 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb2dcff_114f_48d9_9baf_401b6ac59183.slice/crio-33ed5151c66688ff04a9922d759643707c3ecd32fd78979ca9e673ff4ceef1cf WatchSource:0}: Error finding container 33ed5151c66688ff04a9922d759643707c3ecd32fd78979ca9e673ff4ceef1cf: Status 404 returned error can't find the container with id 33ed5151c66688ff04a9922d759643707c3ecd32fd78979ca9e673ff4ceef1cf Sep 14 09:57:54.873791 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.873702 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf"] Sep 14 09:57:54.877721 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.877699 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 09:57:54.880285 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.880267 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-24873d-dockercfg-lrtx2\"" Sep 14 09:57:54.880389 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.880325 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-24873d\"" Sep 14 09:57:54.898293 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.898268 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf"] Sep 14 09:57:54.941552 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:54.941526 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" event={"ID":"afb2dcff-114f-48d9-9baf-401b6ac59183","Type":"ContainerStarted","Data":"33ed5151c66688ff04a9922d759643707c3ecd32fd78979ca9e673ff4ceef1cf"} Sep 14 09:57:55.018348 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.018305 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0283d4d0-2868-4754-9323-53af7d7ded81-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf\" (UID: \"0283d4d0-2868-4754-9323-53af7d7ded81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 09:57:55.018451 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.018362 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zchbl\" (UniqueName: \"kubernetes.io/projected/0283d4d0-2868-4754-9323-53af7d7ded81-kube-api-access-zchbl\") pod \"payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf\" (UID: \"0283d4d0-2868-4754-9323-53af7d7ded81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 09:57:55.119365 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.119330 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0283d4d0-2868-4754-9323-53af7d7ded81-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf\" (UID: \"0283d4d0-2868-4754-9323-53af7d7ded81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 09:57:55.119365 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.119369 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zchbl\" (UniqueName: \"kubernetes.io/projected/0283d4d0-2868-4754-9323-53af7d7ded81-kube-api-access-zchbl\") pod \"payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf\" (UID: \"0283d4d0-2868-4754-9323-53af7d7ded81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 09:57:55.120098 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.120070 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0283d4d0-2868-4754-9323-53af7d7ded81-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf\" (UID: \"0283d4d0-2868-4754-9323-53af7d7ded81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 09:57:55.128452 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.128397 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zchbl\" (UniqueName: \"kubernetes.io/projected/0283d4d0-2868-4754-9323-53af7d7ded81-kube-api-access-zchbl\") pod \"payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf\" (UID: \"0283d4d0-2868-4754-9323-53af7d7ded81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 09:57:55.188932 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.188897 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 09:57:55.470226 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.470198 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf"] Sep 14 09:57:55.472049 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:57:55.472025 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0283d4d0_2868_4754_9323_53af7d7ded81.slice/crio-fbe359074a9ca4405f5e7bbb9eced082ef51fe7c6125855ccbd1075823b712e7 WatchSource:0}: Error finding container fbe359074a9ca4405f5e7bbb9eced082ef51fe7c6125855ccbd1075823b712e7: Status 404 returned error can't find the container with id fbe359074a9ca4405f5e7bbb9eced082ef51fe7c6125855ccbd1075823b712e7 Sep 14 09:57:55.949333 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.949298 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" event={"ID":"0283d4d0-2868-4754-9323-53af7d7ded81","Type":"ContainerStarted","Data":"c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560"} Sep 14 09:57:55.949789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.949366 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" event={"ID":"0283d4d0-2868-4754-9323-53af7d7ded81","Type":"ContainerStarted","Data":"fbe359074a9ca4405f5e7bbb9eced082ef51fe7c6125855ccbd1075823b712e7"} Sep 14 09:57:55.949789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.949397 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 09:57:55.951052 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.951019 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" event={"ID":"afb2dcff-114f-48d9-9baf-401b6ac59183","Type":"ContainerStarted","Data":"7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c"} Sep 14 09:57:55.951165 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.951152 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:57:55.995423 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.995378 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" podStartSLOduration=2.654972487 podStartE2EDuration="2.995363264s" totalImagesPullingTime="340.390777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:55.994065774 +0000 UTC m=+892.689191079" watchObservedRunningTime="2026-09-14 09:57:55.995363264 +0000 UTC m=+892.690488569" Sep 14 09:57:55.995957 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:57:55.995929 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" podStartSLOduration=1.995923451 podStartE2EDuration="1.995923451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:55.971243064 +0000 UTC m=+892.666368368" watchObservedRunningTime="2026-09-14 09:57:55.995923451 +0000 UTC m=+892.691048760" Sep 14 09:58:01.315084 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.315046 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg"] Sep 14 09:58:01.316047 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.315993 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" podUID="29de6996-dcd5-4718-a4cf-c52d7c78e11f" containerName="payload-processing" containerID="cri-o://b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3" gracePeriod=30 Sep 14 09:58:01.327434 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.327403 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5"] Sep 14 09:58:01.327746 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.327692 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" podUID="42379a1a-bbca-410a-8b56-efb32fffd6ad" containerName="payload-pre-processing" containerID="cri-o://c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2" gracePeriod=30 Sep 14 09:58:01.605824 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.605803 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:58:01.608747 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.608726 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:58:01.683819 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.683788 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trldz\" (UniqueName: \"kubernetes.io/projected/42379a1a-bbca-410a-8b56-efb32fffd6ad-kube-api-access-trldz\") pod \"42379a1a-bbca-410a-8b56-efb32fffd6ad\" (UID: \"42379a1a-bbca-410a-8b56-efb32fffd6ad\") " Sep 14 09:58:01.683976 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.683829 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29de6996-dcd5-4718-a4cf-c52d7c78e11f-plugins-config-volume\") pod \"29de6996-dcd5-4718-a4cf-c52d7c78e11f\" (UID: \"29de6996-dcd5-4718-a4cf-c52d7c78e11f\") " Sep 14 09:58:01.683976 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.683880 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42379a1a-bbca-410a-8b56-efb32fffd6ad-plugins-config-volume\") pod \"42379a1a-bbca-410a-8b56-efb32fffd6ad\" (UID: \"42379a1a-bbca-410a-8b56-efb32fffd6ad\") " Sep 14 09:58:01.683976 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.683946 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsjqw\" (UniqueName: \"kubernetes.io/projected/29de6996-dcd5-4718-a4cf-c52d7c78e11f-kube-api-access-nsjqw\") pod \"29de6996-dcd5-4718-a4cf-c52d7c78e11f\" (UID: \"29de6996-dcd5-4718-a4cf-c52d7c78e11f\") " Sep 14 09:58:01.684265 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.684240 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29de6996-dcd5-4718-a4cf-c52d7c78e11f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29de6996-dcd5-4718-a4cf-c52d7c78e11f" (UID: "29de6996-dcd5-4718-a4cf-c52d7c78e11f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 09:58:01.684338 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.684252 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42379a1a-bbca-410a-8b56-efb32fffd6ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42379a1a-bbca-410a-8b56-efb32fffd6ad" (UID: "42379a1a-bbca-410a-8b56-efb32fffd6ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 09:58:01.686031 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.686012 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42379a1a-bbca-410a-8b56-efb32fffd6ad-kube-api-access-trldz" (OuterVolumeSpecName: "kube-api-access-trldz") pod "42379a1a-bbca-410a-8b56-efb32fffd6ad" (UID: "42379a1a-bbca-410a-8b56-efb32fffd6ad"). InnerVolumeSpecName "kube-api-access-trldz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 09:58:01.686153 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.686131 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29de6996-dcd5-4718-a4cf-c52d7c78e11f-kube-api-access-nsjqw" (OuterVolumeSpecName: "kube-api-access-nsjqw") pod "29de6996-dcd5-4718-a4cf-c52d7c78e11f" (UID: "29de6996-dcd5-4718-a4cf-c52d7c78e11f"). InnerVolumeSpecName "kube-api-access-nsjqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 09:58:01.784942 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.784906 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42379a1a-bbca-410a-8b56-efb32fffd6ad-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:58:01.784942 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.784934 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsjqw\" (UniqueName: \"kubernetes.io/projected/29de6996-dcd5-4718-a4cf-c52d7c78e11f-kube-api-access-nsjqw\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:58:01.784942 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.784943 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trldz\" (UniqueName: \"kubernetes.io/projected/42379a1a-bbca-410a-8b56-efb32fffd6ad-kube-api-access-trldz\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:58:01.785167 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.784953 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29de6996-dcd5-4718-a4cf-c52d7c78e11f-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:58:01.962439 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.962410 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 09:58:01.984789 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.984756 2550 generic.go:356] "Generic (PLEG): container finished" podID="29de6996-dcd5-4718-a4cf-c52d7c78e11f" containerID="b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3" exitCode=0 Sep 14 09:58:01.984971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.984798 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" Sep 14 09:58:01.984971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.984847 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" event={"ID":"29de6996-dcd5-4718-a4cf-c52d7c78e11f","Type":"ContainerDied","Data":"b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3"} Sep 14 09:58:01.984971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.984894 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg" event={"ID":"29de6996-dcd5-4718-a4cf-c52d7c78e11f","Type":"ContainerDied","Data":"72c63b470b4577743baf7e82297f1ba0fecbb4be95e7f2a15d93f9b1922ecdf3"} Sep 14 09:58:01.984971 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.984922 2550 scope.go:117] "RemoveContainer" containerID="b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3" Sep 14 09:58:01.986524 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.986496 2550 generic.go:356] "Generic (PLEG): container finished" podID="42379a1a-bbca-410a-8b56-efb32fffd6ad" containerID="c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2" exitCode=0 Sep 14 09:58:01.986619 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.986544 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" event={"ID":"42379a1a-bbca-410a-8b56-efb32fffd6ad","Type":"ContainerDied","Data":"c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2"} Sep 14 09:58:01.986619 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.986583 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" event={"ID":"42379a1a-bbca-410a-8b56-efb32fffd6ad","Type":"ContainerDied","Data":"2f7afc72b2322ac4adcb8614b56cd51124ec9371aafe482ccccad958fcda59c5"} Sep 14 09:58:01.986619 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.986548 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5" Sep 14 09:58:01.995385 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.995366 2550 scope.go:117] "RemoveContainer" containerID="b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3" Sep 14 09:58:01.995753 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:58:01.995718 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3\": container with ID starting with b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3 not found: ID does not exist" containerID="b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3" Sep 14 09:58:01.995847 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.995762 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3"} err="failed to get container status \"b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3\": rpc error: code = NotFound desc = could not find container \"b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3\": container with ID starting with b698c115288634c8c762ae2546ce331a75fa17f70760a091e6294a867253f8f3 not found: ID does not exist" Sep 14 09:58:01.995847 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:01.995786 2550 scope.go:117] "RemoveContainer" containerID="c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2" Sep 14 09:58:02.007283 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:02.007262 2550 scope.go:117] "RemoveContainer" containerID="c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2" Sep 14 09:58:02.007586 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:58:02.007563 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2\": container with ID starting with c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2 not found: ID does not exist" containerID="c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2" Sep 14 09:58:02.007674 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:02.007598 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2"} err="failed to get container status \"c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2\": rpc error: code = NotFound desc = could not find container \"c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2\": container with ID starting with c86df91a160df4ae03b335ae279d062240a182db6ab018e97f0508e053317db2 not found: ID does not exist" Sep 14 09:58:02.012434 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:02.012414 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5"] Sep 14 09:58:02.022023 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:02.017335 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9eed367b-6878999c74-qkgm5"] Sep 14 09:58:02.028898 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:02.028876 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg"] Sep 14 09:58:02.034877 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:02.034855 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9eed367b-56d7f75465-8kskg"] Sep 14 09:58:03.827743 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:03.827605 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 09:58:03.827743 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:03.827685 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29de6996-dcd5-4718-a4cf-c52d7c78e11f" path="/var/lib/kubelet/pods/29de6996-dcd5-4718-a4cf-c52d7c78e11f/volumes" Sep 14 09:58:03.828181 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:03.828167 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42379a1a-bbca-410a-8b56-efb32fffd6ad" path="/var/lib/kubelet/pods/42379a1a-bbca-410a-8b56-efb32fffd6ad/volumes" Sep 14 09:58:03.829443 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:03.829421 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 09:58:06.959022 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:06.958989 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 09:58:08.220918 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.220877 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j"] Sep 14 09:58:08.221388 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.221320 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42379a1a-bbca-410a-8b56-efb32fffd6ad" containerName="payload-pre-processing" Sep 14 09:58:08.221388 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.221331 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="42379a1a-bbca-410a-8b56-efb32fffd6ad" containerName="payload-pre-processing" Sep 14 09:58:08.221388 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.221342 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29de6996-dcd5-4718-a4cf-c52d7c78e11f" containerName="payload-processing" Sep 14 09:58:08.221388 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.221348 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="29de6996-dcd5-4718-a4cf-c52d7c78e11f" containerName="payload-processing" Sep 14 09:58:08.221594 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.221417 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="29de6996-dcd5-4718-a4cf-c52d7c78e11f" containerName="payload-processing" Sep 14 09:58:08.221594 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.221427 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="42379a1a-bbca-410a-8b56-efb32fffd6ad" containerName="payload-pre-processing" Sep 14 09:58:08.226253 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.226233 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:08.228784 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.228759 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2f4712d0\"" Sep 14 09:58:08.228912 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.228859 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2f4712d0-dockercfg-96dht\"" Sep 14 09:58:08.235378 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.235354 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j"] Sep 14 09:58:08.250084 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.250060 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bca9db-260b-4136-8d9d-5825e907b536-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j\" (UID: \"08bca9db-260b-4136-8d9d-5825e907b536\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:08.250229 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.250211 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwsd\" (UniqueName: \"kubernetes.io/projected/08bca9db-260b-4136-8d9d-5825e907b536-kube-api-access-zxwsd\") pod \"payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j\" (UID: \"08bca9db-260b-4136-8d9d-5825e907b536\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:08.326160 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.326125 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls"] Sep 14 09:58:08.330944 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.330857 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:08.340954 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.340919 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls"] Sep 14 09:58:08.350933 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.350902 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2x4l\" (UniqueName: \"kubernetes.io/projected/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-kube-api-access-b2x4l\") pod \"payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls\" (UID: \"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191\") " pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:08.351056 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.351035 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwsd\" (UniqueName: \"kubernetes.io/projected/08bca9db-260b-4136-8d9d-5825e907b536-kube-api-access-zxwsd\") pod \"payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j\" (UID: \"08bca9db-260b-4136-8d9d-5825e907b536\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:08.351155 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.351135 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bca9db-260b-4136-8d9d-5825e907b536-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j\" (UID: \"08bca9db-260b-4136-8d9d-5825e907b536\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:08.351216 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.351174 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls\" (UID: \"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191\") " pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:08.352063 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.352039 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bca9db-260b-4136-8d9d-5825e907b536-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j\" (UID: \"08bca9db-260b-4136-8d9d-5825e907b536\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:08.359613 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.359586 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwsd\" (UniqueName: \"kubernetes.io/projected/08bca9db-260b-4136-8d9d-5825e907b536-kube-api-access-zxwsd\") pod \"payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j\" (UID: \"08bca9db-260b-4136-8d9d-5825e907b536\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:08.451845 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.451817 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls\" (UID: \"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191\") " pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:08.451985 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.451886 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2x4l\" (UniqueName: \"kubernetes.io/projected/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-kube-api-access-b2x4l\") pod \"payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls\" (UID: \"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191\") " pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:08.452516 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.452496 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls\" (UID: \"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191\") " pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:08.460773 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.460751 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2x4l\" (UniqueName: \"kubernetes.io/projected/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-kube-api-access-b2x4l\") pod \"payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls\" (UID: \"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191\") " pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:08.537885 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.537822 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:08.644286 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.644251 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:08.809915 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.809892 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j"] Sep 14 09:58:08.811084 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:58:08.811062 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bca9db_260b_4136_8d9d_5825e907b536.slice/crio-1d91e389271275e5e63cb70d57b78ef8ea274446d2f40d87a6ff3b68f1019566 WatchSource:0}: Error finding container 1d91e389271275e5e63cb70d57b78ef8ea274446d2f40d87a6ff3b68f1019566: Status 404 returned error can't find the container with id 1d91e389271275e5e63cb70d57b78ef8ea274446d2f40d87a6ff3b68f1019566 Sep 14 09:58:08.922278 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:08.922241 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls"] Sep 14 09:58:08.923359 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:58:08.923333 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39c4bc6_ac21_4ce6_adbe_8c73c4f23191.slice/crio-8f9763b59a91d8a3655c4666ee55bae3f1058b2efbee391708495775094f758c WatchSource:0}: Error finding container 8f9763b59a91d8a3655c4666ee55bae3f1058b2efbee391708495775094f758c: Status 404 returned error can't find the container with id 8f9763b59a91d8a3655c4666ee55bae3f1058b2efbee391708495775094f758c Sep 14 09:58:09.032528 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:09.032493 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" event={"ID":"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191","Type":"ContainerStarted","Data":"c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520"} Sep 14 09:58:09.032765 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:09.032556 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" event={"ID":"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191","Type":"ContainerStarted","Data":"8f9763b59a91d8a3655c4666ee55bae3f1058b2efbee391708495775094f758c"} Sep 14 09:58:09.032765 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:09.032752 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:09.034143 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:09.034117 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" event={"ID":"08bca9db-260b-4136-8d9d-5825e907b536","Type":"ContainerStarted","Data":"163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe"} Sep 14 09:58:09.034245 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:09.034163 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:09.034245 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:09.034183 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" event={"ID":"08bca9db-260b-4136-8d9d-5825e907b536","Type":"ContainerStarted","Data":"1d91e389271275e5e63cb70d57b78ef8ea274446d2f40d87a6ff3b68f1019566"} Sep 14 09:58:09.053216 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:09.053164 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" podStartSLOduration=1.053145684 podStartE2EDuration="1.053145684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:58:09.049782779 +0000 UTC m=+905.744908084" watchObservedRunningTime="2026-09-14 09:58:09.053145684 +0000 UTC m=+905.748270991" Sep 14 09:58:09.068084 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:09.068008 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" podStartSLOduration=1.06799564 podStartE2EDuration="1.06799564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:58:09.06587891 +0000 UTC m=+905.761004216" watchObservedRunningTime="2026-09-14 09:58:09.06799564 +0000 UTC m=+905.763120991" Sep 14 09:58:20.043237 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:20.043202 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:20.043738 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:20.043341 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:37.102231 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.102195 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls"] Sep 14 09:58:37.102863 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.102455 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" podUID="f39c4bc6-ac21-4ce6-adbe-8c73c4f23191" containerName="payload-processing" containerID="cri-o://c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520" gracePeriod=30 Sep 14 09:58:37.115802 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.115776 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j"] Sep 14 09:58:37.116193 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.116144 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" podUID="08bca9db-260b-4136-8d9d-5825e907b536" containerName="payload-pre-processing" containerID="cri-o://163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe" gracePeriod=30 Sep 14 09:58:37.412379 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.412358 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:37.415233 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.415209 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:37.445921 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.445894 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bca9db-260b-4136-8d9d-5825e907b536-plugins-config-volume\") pod \"08bca9db-260b-4136-8d9d-5825e907b536\" (UID: \"08bca9db-260b-4136-8d9d-5825e907b536\") " Sep 14 09:58:37.446083 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.445951 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxwsd\" (UniqueName: \"kubernetes.io/projected/08bca9db-260b-4136-8d9d-5825e907b536-kube-api-access-zxwsd\") pod \"08bca9db-260b-4136-8d9d-5825e907b536\" (UID: \"08bca9db-260b-4136-8d9d-5825e907b536\") " Sep 14 09:58:37.446083 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.446046 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-plugins-config-volume\") pod \"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191\" (UID: \"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191\") " Sep 14 09:58:37.446248 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.446098 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2x4l\" (UniqueName: \"kubernetes.io/projected/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-kube-api-access-b2x4l\") pod \"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191\" (UID: \"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191\") " Sep 14 09:58:37.446889 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.446293 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bca9db-260b-4136-8d9d-5825e907b536-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08bca9db-260b-4136-8d9d-5825e907b536" (UID: "08bca9db-260b-4136-8d9d-5825e907b536"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 09:58:37.446889 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.446470 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f39c4bc6-ac21-4ce6-adbe-8c73c4f23191" (UID: "f39c4bc6-ac21-4ce6-adbe-8c73c4f23191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 09:58:37.446889 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.446628 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:58:37.446889 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.446664 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bca9db-260b-4136-8d9d-5825e907b536-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:58:37.449156 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.449107 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-kube-api-access-b2x4l" (OuterVolumeSpecName: "kube-api-access-b2x4l") pod "f39c4bc6-ac21-4ce6-adbe-8c73c4f23191" (UID: "f39c4bc6-ac21-4ce6-adbe-8c73c4f23191"). InnerVolumeSpecName "kube-api-access-b2x4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 09:58:37.449264 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.449235 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bca9db-260b-4136-8d9d-5825e907b536-kube-api-access-zxwsd" (OuterVolumeSpecName: "kube-api-access-zxwsd") pod "08bca9db-260b-4136-8d9d-5825e907b536" (UID: "08bca9db-260b-4136-8d9d-5825e907b536"). InnerVolumeSpecName "kube-api-access-zxwsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 09:58:37.547995 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.547956 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2x4l\" (UniqueName: \"kubernetes.io/projected/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191-kube-api-access-b2x4l\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:58:37.547995 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:37.547991 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwsd\" (UniqueName: \"kubernetes.io/projected/08bca9db-260b-4136-8d9d-5825e907b536-kube-api-access-zxwsd\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:58:38.192879 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.192840 2550 generic.go:356] "Generic (PLEG): container finished" podID="f39c4bc6-ac21-4ce6-adbe-8c73c4f23191" containerID="c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520" exitCode=0 Sep 14 09:58:38.193341 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.192900 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" Sep 14 09:58:38.193592 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.192941 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" event={"ID":"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191","Type":"ContainerDied","Data":"c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520"} Sep 14 09:58:38.193702 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.193630 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls" event={"ID":"f39c4bc6-ac21-4ce6-adbe-8c73c4f23191","Type":"ContainerDied","Data":"8f9763b59a91d8a3655c4666ee55bae3f1058b2efbee391708495775094f758c"} Sep 14 09:58:38.193702 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.193672 2550 scope.go:117] "RemoveContainer" containerID="c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520" Sep 14 09:58:38.195158 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.194918 2550 generic.go:356] "Generic (PLEG): container finished" podID="08bca9db-260b-4136-8d9d-5825e907b536" containerID="163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe" exitCode=0 Sep 14 09:58:38.195158 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.195037 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" Sep 14 09:58:38.195158 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.195078 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" event={"ID":"08bca9db-260b-4136-8d9d-5825e907b536","Type":"ContainerDied","Data":"163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe"} Sep 14 09:58:38.195158 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.195113 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j" event={"ID":"08bca9db-260b-4136-8d9d-5825e907b536","Type":"ContainerDied","Data":"1d91e389271275e5e63cb70d57b78ef8ea274446d2f40d87a6ff3b68f1019566"} Sep 14 09:58:38.206731 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.206711 2550 scope.go:117] "RemoveContainer" containerID="c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520" Sep 14 09:58:38.207342 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:58:38.207316 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520\": container with ID starting with c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520 not found: ID does not exist" containerID="c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520" Sep 14 09:58:38.207435 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.207353 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520"} err="failed to get container status \"c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520\": rpc error: code = NotFound desc = could not find container \"c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520\": container with ID starting with c80f84c27a5d3b43bb8537ce2e620cd01912b4d5173a14541d1d942169c89520 not found: ID does not exist" Sep 14 09:58:38.207435 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.207376 2550 scope.go:117] "RemoveContainer" containerID="163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe" Sep 14 09:58:38.217455 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.217427 2550 scope.go:117] "RemoveContainer" containerID="163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe" Sep 14 09:58:38.217878 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:58:38.217849 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe\": container with ID starting with 163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe not found: ID does not exist" containerID="163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe" Sep 14 09:58:38.217979 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.217879 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe"} err="failed to get container status \"163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe\": rpc error: code = NotFound desc = could not find container \"163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe\": container with ID starting with 163888e2b3f7198f2991d97b70839fc9cf70d6e42d705bcc807ab9156d4da8fe not found: ID does not exist" Sep 14 09:58:38.230317 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.230295 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j"] Sep 14 09:58:38.239372 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.239348 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2f4712d0-7f7bc5d766-hfg9j"] Sep 14 09:58:38.255511 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.255487 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls"] Sep 14 09:58:38.265164 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:38.264072 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2f4712d0-7b66f5b99d-hdbls"] Sep 14 09:58:39.824976 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:39.824944 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08bca9db-260b-4136-8d9d-5825e907b536" path="/var/lib/kubelet/pods/08bca9db-260b-4136-8d9d-5825e907b536/volumes" Sep 14 09:58:39.825351 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:39.825318 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f39c4bc6-ac21-4ce6-adbe-8c73c4f23191" path="/var/lib/kubelet/pods/f39c4bc6-ac21-4ce6-adbe-8c73c4f23191/volumes" Sep 14 09:58:45.150254 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.150213 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl"] Sep 14 09:58:45.152815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.150851 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39c4bc6-ac21-4ce6-adbe-8c73c4f23191" containerName="payload-processing" Sep 14 09:58:45.152815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.150869 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39c4bc6-ac21-4ce6-adbe-8c73c4f23191" containerName="payload-processing" Sep 14 09:58:45.152815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.150884 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08bca9db-260b-4136-8d9d-5825e907b536" containerName="payload-pre-processing" Sep 14 09:58:45.152815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.150890 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="08bca9db-260b-4136-8d9d-5825e907b536" containerName="payload-pre-processing" Sep 14 09:58:45.152815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.150960 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="08bca9db-260b-4136-8d9d-5825e907b536" containerName="payload-pre-processing" Sep 14 09:58:45.152815 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.150969 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f39c4bc6-ac21-4ce6-adbe-8c73c4f23191" containerName="payload-processing" Sep 14 09:58:45.154091 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.154072 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:58:45.156866 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.156843 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e7fe0e18\"" Sep 14 09:58:45.168679 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.168653 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl"] Sep 14 09:58:45.223761 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.223724 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl\" (UID: \"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:58:45.223933 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.223836 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhzz\" (UniqueName: \"kubernetes.io/projected/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-kube-api-access-7bhzz\") pod \"payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl\" (UID: \"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:58:45.325272 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.325239 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhzz\" (UniqueName: \"kubernetes.io/projected/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-kube-api-access-7bhzz\") pod \"payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl\" (UID: \"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:58:45.325461 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.325326 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl\" (UID: \"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:58:45.325980 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.325961 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl\" (UID: \"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:58:45.340113 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.340087 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhzz\" (UniqueName: \"kubernetes.io/projected/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-kube-api-access-7bhzz\") pod \"payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl\" (UID: \"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:58:45.465330 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.465299 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:58:45.742162 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:45.742136 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl"] Sep 14 09:58:45.744593 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:58:45.744567 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8384a2_5ff1_4b7f_a9a5_a7db4889c027.slice/crio-e3ffc5e8f1576a795105417a744f1a213337b179a76008978efec732bc42b70c WatchSource:0}: Error finding container e3ffc5e8f1576a795105417a744f1a213337b179a76008978efec732bc42b70c: Status 404 returned error can't find the container with id e3ffc5e8f1576a795105417a744f1a213337b179a76008978efec732bc42b70c Sep 14 09:58:46.242899 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:46.242864 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" event={"ID":"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027","Type":"ContainerStarted","Data":"2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c"} Sep 14 09:58:46.243327 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:46.243047 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:58:46.243327 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:46.243079 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" event={"ID":"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027","Type":"ContainerStarted","Data":"e3ffc5e8f1576a795105417a744f1a213337b179a76008978efec732bc42b70c"} Sep 14 09:58:46.275691 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:46.275628 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" podStartSLOduration=1.2756131640000001 podStartE2EDuration="1.275613164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:58:46.273732048 +0000 UTC m=+942.968857352" watchObservedRunningTime="2026-09-14 09:58:46.275613164 +0000 UTC m=+942.970738526" Sep 14 09:58:57.251419 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:58:57.251386 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:59:23.382337 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:23.382299 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl"] Sep 14 09:59:23.382949 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:23.382589 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" podUID="fc8384a2-5ff1-4b7f-a9a5-a7db4889c027" containerName="payload-pre-processing" containerID="cri-o://2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c" gracePeriod=30 Sep 14 09:59:23.654421 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:23.654394 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:59:23.696044 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:23.696008 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bhzz\" (UniqueName: \"kubernetes.io/projected/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-kube-api-access-7bhzz\") pod \"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027\" (UID: \"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027\") " Sep 14 09:59:23.696245 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:23.696088 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-plugins-config-volume\") pod \"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027\" (UID: \"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027\") " Sep 14 09:59:23.696750 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:23.696721 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc8384a2-5ff1-4b7f-a9a5-a7db4889c027" (UID: "fc8384a2-5ff1-4b7f-a9a5-a7db4889c027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 09:59:23.698073 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:23.698050 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-kube-api-access-7bhzz" (OuterVolumeSpecName: "kube-api-access-7bhzz") pod "fc8384a2-5ff1-4b7f-a9a5-a7db4889c027" (UID: "fc8384a2-5ff1-4b7f-a9a5-a7db4889c027"). InnerVolumeSpecName "kube-api-access-7bhzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 09:59:23.797431 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:23.797388 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bhzz\" (UniqueName: \"kubernetes.io/projected/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-kube-api-access-7bhzz\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:59:23.797431 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:23.797426 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 09:59:24.451293 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:24.451256 2550 generic.go:356] "Generic (PLEG): container finished" podID="fc8384a2-5ff1-4b7f-a9a5-a7db4889c027" containerID="2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c" exitCode=0 Sep 14 09:59:24.451865 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:24.451323 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" Sep 14 09:59:24.451865 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:24.451349 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" event={"ID":"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027","Type":"ContainerDied","Data":"2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c"} Sep 14 09:59:24.451865 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:24.451406 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl" event={"ID":"fc8384a2-5ff1-4b7f-a9a5-a7db4889c027","Type":"ContainerDied","Data":"e3ffc5e8f1576a795105417a744f1a213337b179a76008978efec732bc42b70c"} Sep 14 09:59:24.451865 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:24.451433 2550 scope.go:117] "RemoveContainer" containerID="2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c" Sep 14 09:59:24.460478 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:24.460461 2550 scope.go:117] "RemoveContainer" containerID="2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c" Sep 14 09:59:24.460868 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:59:24.460845 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c\": container with ID starting with 2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c not found: ID does not exist" containerID="2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c" Sep 14 09:59:24.460959 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:24.460872 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c"} err="failed to get container status \"2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c\": rpc error: code = NotFound desc = could not find container \"2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c\": container with ID starting with 2beabb2f9cf518be9d202757711d62ff213477a55e60b92c99b541e92e60b96c not found: ID does not exist" Sep 14 09:59:24.469869 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:24.469843 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl"] Sep 14 09:59:24.471825 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:24.471804 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7fe0e18-c798bbcdb-l2wpl"] Sep 14 09:59:25.824507 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:25.824473 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8384a2-5ff1-4b7f-a9a5-a7db4889c027" path="/var/lib/kubelet/pods/fc8384a2-5ff1-4b7f-a9a5-a7db4889c027/volumes" Sep 14 09:59:30.671426 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.670991 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9"] Sep 14 09:59:30.672109 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.672091 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8384a2-5ff1-4b7f-a9a5-a7db4889c027" containerName="payload-pre-processing" Sep 14 09:59:30.672190 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.672111 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8384a2-5ff1-4b7f-a9a5-a7db4889c027" containerName="payload-pre-processing" Sep 14 09:59:30.672225 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.672213 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8384a2-5ff1-4b7f-a9a5-a7db4889c027" containerName="payload-pre-processing" Sep 14 09:59:30.675550 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.675529 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.680884 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.680863 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-80fa7537-serving-cert\"" Sep 14 09:59:30.689805 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.689781 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9"] Sep 14 09:59:30.767677 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.767648 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-metrics-certs\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.767868 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.767708 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-maas-api-tls\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.767868 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.767741 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cda49c-5891-496b-8f2a-028c8b0fb421-openshift-service-ca\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.767868 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.767762 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnpx\" (UniqueName: \"kubernetes.io/projected/b7cda49c-5891-496b-8f2a-028c8b0fb421-kube-api-access-zjnpx\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.868524 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.868491 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-metrics-certs\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.868749 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.868543 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-maas-api-tls\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.868749 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.868561 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cda49c-5891-496b-8f2a-028c8b0fb421-openshift-service-ca\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.868749 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.868585 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnpx\" (UniqueName: \"kubernetes.io/projected/b7cda49c-5891-496b-8f2a-028c8b0fb421-kube-api-access-zjnpx\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.868749 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:59:30.868699 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-serving-cert: secret "maas-api-e2e-derive-80fa7537-serving-cert" not found Sep 14 09:59:30.868977 ip-10-0-135-40 kubenswrapper[2550]: E0914 09:59:30.868784 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-maas-api-tls podName:b7cda49c-5891-496b-8f2a-028c8b0fb421 nodeName:}" failed. No retries permitted until 2026-09-14 09:59:31.368763476 +0000 UTC m=+988.063888765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-maas-api-tls") pod "maas-api-e2e-derive-80fa7537-5949db8574-jtff9" (UID: "b7cda49c-5891-496b-8f2a-028c8b0fb421") : secret "maas-api-e2e-derive-80fa7537-serving-cert" not found Sep 14 09:59:30.869172 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.869154 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cda49c-5891-496b-8f2a-028c8b0fb421-openshift-service-ca\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.871018 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.870982 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-metrics-certs\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:30.881805 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:30.881774 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnpx\" (UniqueName: \"kubernetes.io/projected/b7cda49c-5891-496b-8f2a-028c8b0fb421-kube-api-access-zjnpx\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:31.373914 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.373882 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-maas-api-tls\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:31.376171 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.376147 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-maas-api-tls\") pod \"maas-api-e2e-derive-80fa7537-5949db8574-jtff9\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:31.552308 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.552276 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx"] Sep 14 09:59:31.555903 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.555887 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 09:59:31.558441 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.558418 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-80fa7537\"" Sep 14 09:59:31.558566 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.558475 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-80fa7537-dockercfg-clhbx\"" Sep 14 09:59:31.566902 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.566881 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx"] Sep 14 09:59:31.586286 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.586264 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:31.677510 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.677475 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4d7cba-9c8e-4710-b562-1333b413e65c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx\" (UID: \"2f4d7cba-9c8e-4710-b562-1333b413e65c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 09:59:31.677957 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.677573 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhc9w\" (UniqueName: \"kubernetes.io/projected/2f4d7cba-9c8e-4710-b562-1333b413e65c-kube-api-access-mhc9w\") pod \"payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx\" (UID: \"2f4d7cba-9c8e-4710-b562-1333b413e65c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 09:59:31.778486 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.778459 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4d7cba-9c8e-4710-b562-1333b413e65c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx\" (UID: \"2f4d7cba-9c8e-4710-b562-1333b413e65c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 09:59:31.778646 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.778502 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc9w\" (UniqueName: \"kubernetes.io/projected/2f4d7cba-9c8e-4710-b562-1333b413e65c-kube-api-access-mhc9w\") pod \"payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx\" (UID: \"2f4d7cba-9c8e-4710-b562-1333b413e65c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 09:59:31.779207 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.779187 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4d7cba-9c8e-4710-b562-1333b413e65c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx\" (UID: \"2f4d7cba-9c8e-4710-b562-1333b413e65c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 09:59:31.787565 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.787544 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhc9w\" (UniqueName: \"kubernetes.io/projected/2f4d7cba-9c8e-4710-b562-1333b413e65c-kube-api-access-mhc9w\") pod \"payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx\" (UID: \"2f4d7cba-9c8e-4710-b562-1333b413e65c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 09:59:31.867415 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.867393 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 09:59:31.868559 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:31.868533 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9"] Sep 14 09:59:31.869689 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:59:31.869663 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cda49c_5891_496b_8f2a_028c8b0fb421.slice/crio-6d08286b3a953c8f0fd272990f833b9f2ea65e3a4ed5d999cbfe793dca29f056 WatchSource:0}: Error finding container 6d08286b3a953c8f0fd272990f833b9f2ea65e3a4ed5d999cbfe793dca29f056: Status 404 returned error can't find the container with id 6d08286b3a953c8f0fd272990f833b9f2ea65e3a4ed5d999cbfe793dca29f056 Sep 14 09:59:32.158278 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:32.158255 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx"] Sep 14 09:59:32.159995 ip-10-0-135-40 kubenswrapper[2550]: W0914 09:59:32.159970 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4d7cba_9c8e_4710_b562_1333b413e65c.slice/crio-fd14fb644515538faf25757350e61154e704bd50c4ab730c4c11992c85035d9d WatchSource:0}: Error finding container fd14fb644515538faf25757350e61154e704bd50c4ab730c4c11992c85035d9d: Status 404 returned error can't find the container with id fd14fb644515538faf25757350e61154e704bd50c4ab730c4c11992c85035d9d Sep 14 09:59:32.495793 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:32.495701 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" event={"ID":"2f4d7cba-9c8e-4710-b562-1333b413e65c","Type":"ContainerStarted","Data":"e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084"} Sep 14 09:59:32.495949 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:32.495793 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" event={"ID":"2f4d7cba-9c8e-4710-b562-1333b413e65c","Type":"ContainerStarted","Data":"fd14fb644515538faf25757350e61154e704bd50c4ab730c4c11992c85035d9d"} Sep 14 09:59:32.495949 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:32.495929 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 09:59:32.497447 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:32.497415 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" event={"ID":"b7cda49c-5891-496b-8f2a-028c8b0fb421","Type":"ContainerStarted","Data":"1617110ca041c12a6101e8682dd783ada92c95d4c7c304891423f49f1458ef0a"} Sep 14 09:59:32.497567 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:32.497471 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" event={"ID":"b7cda49c-5891-496b-8f2a-028c8b0fb421","Type":"ContainerStarted","Data":"6d08286b3a953c8f0fd272990f833b9f2ea65e3a4ed5d999cbfe793dca29f056"} Sep 14 09:59:32.497567 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:32.497500 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:32.531600 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:32.531540 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" podStartSLOduration=1.531520477 podStartE2EDuration="1.531520477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:59:32.529161895 +0000 UTC m=+989.224287201" watchObservedRunningTime="2026-09-14 09:59:32.531520477 +0000 UTC m=+989.226645844" Sep 14 09:59:32.558858 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:32.558809 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" podStartSLOduration=2.135928737 podStartE2EDuration="2.558793787s" totalImagesPullingTime="422.86505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:59:32.557876853 +0000 UTC m=+989.253002161" watchObservedRunningTime="2026-09-14 09:59:32.558793787 +0000 UTC m=+989.253919092" Sep 14 09:59:38.508864 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:38.508832 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 09:59:43.505742 ip-10-0-135-40 kubenswrapper[2550]: I0914 09:59:43.505714 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 10:00:00.177531 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.177499 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk"] Sep 14 10:00:00.181595 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.181579 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk" Sep 14 10:00:00.187498 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:00:00.187460 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk\" is forbidden: User \"system:node:ip-10-0-135-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-135-40.ec2.internal' and this object" podUID="95db1855-1a94-47a1-a5a6-856b8a49ff5c" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk" Sep 14 10:00:00.187666 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:00:00.187496 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-cleanup-dockercfg-gf4xj\" is forbidden: User \"system:node:ip-10-0-135-40.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-135-40.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\"" type="*v1.Secret" Sep 14 10:00:00.206729 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.206703 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk"] Sep 14 10:00:00.520271 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.520184 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9"] Sep 14 10:00:00.520456 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.520429 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" podUID="b7cda49c-5891-496b-8f2a-028c8b0fb421" containerName="maas-api" containerID="cri-o://1617110ca041c12a6101e8682dd783ada92c95d4c7c304891423f49f1458ef0a" gracePeriod=30 Sep 14 10:00:00.626026 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.625989 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx"] Sep 14 10:00:00.626295 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.626246 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" podUID="2f4d7cba-9c8e-4710-b562-1333b413e65c" containerName="payload-pre-processing" containerID="cri-o://e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084" gracePeriod=30 Sep 14 10:00:00.654144 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.654098 2550 generic.go:356] "Generic (PLEG): container finished" podID="b7cda49c-5891-496b-8f2a-028c8b0fb421" containerID="1617110ca041c12a6101e8682dd783ada92c95d4c7c304891423f49f1458ef0a" exitCode=0 Sep 14 10:00:00.654244 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.654166 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" event={"ID":"b7cda49c-5891-496b-8f2a-028c8b0fb421","Type":"ContainerDied","Data":"1617110ca041c12a6101e8682dd783ada92c95d4c7c304891423f49f1458ef0a"} Sep 14 10:00:00.785444 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.785383 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 10:00:00.858694 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.858667 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjnpx\" (UniqueName: \"kubernetes.io/projected/b7cda49c-5891-496b-8f2a-028c8b0fb421-kube-api-access-zjnpx\") pod \"b7cda49c-5891-496b-8f2a-028c8b0fb421\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " Sep 14 10:00:00.858859 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.858723 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-maas-api-tls\") pod \"b7cda49c-5891-496b-8f2a-028c8b0fb421\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " Sep 14 10:00:00.858921 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.858863 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-metrics-certs\") pod \"b7cda49c-5891-496b-8f2a-028c8b0fb421\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " Sep 14 10:00:00.858978 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.858963 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cda49c-5891-496b-8f2a-028c8b0fb421-openshift-service-ca\") pod \"b7cda49c-5891-496b-8f2a-028c8b0fb421\" (UID: \"b7cda49c-5891-496b-8f2a-028c8b0fb421\") " Sep 14 10:00:00.859360 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.859326 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7cda49c-5891-496b-8f2a-028c8b0fb421-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7cda49c-5891-496b-8f2a-028c8b0fb421" (UID: "b7cda49c-5891-496b-8f2a-028c8b0fb421"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:00:00.861607 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.861579 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b7cda49c-5891-496b-8f2a-028c8b0fb421" (UID: "b7cda49c-5891-496b-8f2a-028c8b0fb421"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:00:00.861751 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.861601 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7cda49c-5891-496b-8f2a-028c8b0fb421" (UID: "b7cda49c-5891-496b-8f2a-028c8b0fb421"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:00:00.861812 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.861765 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7cda49c-5891-496b-8f2a-028c8b0fb421-kube-api-access-zjnpx" (OuterVolumeSpecName: "kube-api-access-zjnpx") pod "b7cda49c-5891-496b-8f2a-028c8b0fb421" (UID: "b7cda49c-5891-496b-8f2a-028c8b0fb421"). InnerVolumeSpecName "kube-api-access-zjnpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:00:00.886815 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.886796 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 10:00:00.959858 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.959829 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhc9w\" (UniqueName: \"kubernetes.io/projected/2f4d7cba-9c8e-4710-b562-1333b413e65c-kube-api-access-mhc9w\") pod \"2f4d7cba-9c8e-4710-b562-1333b413e65c\" (UID: \"2f4d7cba-9c8e-4710-b562-1333b413e65c\") " Sep 14 10:00:00.960030 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.959921 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4d7cba-9c8e-4710-b562-1333b413e65c-plugins-config-volume\") pod \"2f4d7cba-9c8e-4710-b562-1333b413e65c\" (UID: \"2f4d7cba-9c8e-4710-b562-1333b413e65c\") " Sep 14 10:00:00.960201 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.960184 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cda49c-5891-496b-8f2a-028c8b0fb421-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:00:00.960201 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.960201 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjnpx\" (UniqueName: \"kubernetes.io/projected/b7cda49c-5891-496b-8f2a-028c8b0fb421-kube-api-access-zjnpx\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:00:00.960370 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.960213 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:00:00.960370 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.960227 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cda49c-5891-496b-8f2a-028c8b0fb421-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:00:00.960464 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.960382 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4d7cba-9c8e-4710-b562-1333b413e65c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f4d7cba-9c8e-4710-b562-1333b413e65c" (UID: "2f4d7cba-9c8e-4710-b562-1333b413e65c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:00:00.962174 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:00.962149 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4d7cba-9c8e-4710-b562-1333b413e65c-kube-api-access-mhc9w" (OuterVolumeSpecName: "kube-api-access-mhc9w") pod "2f4d7cba-9c8e-4710-b562-1333b413e65c" (UID: "2f4d7cba-9c8e-4710-b562-1333b413e65c"). InnerVolumeSpecName "kube-api-access-mhc9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:00:01.061189 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.061115 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhc9w\" (UniqueName: \"kubernetes.io/projected/2f4d7cba-9c8e-4710-b562-1333b413e65c-kube-api-access-mhc9w\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:00:01.061189 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.061143 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4d7cba-9c8e-4710-b562-1333b413e65c-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:00:01.192821 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.192793 2550 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk" secret="" err="failed to sync secret cache: timed out waiting for the condition" Sep 14 10:00:01.193199 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.192839 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk" Sep 14 10:00:01.337934 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.337860 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr"] Sep 14 10:00:01.338350 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.338337 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7cda49c-5891-496b-8f2a-028c8b0fb421" containerName="maas-api" Sep 14 10:00:01.338393 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.338352 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7cda49c-5891-496b-8f2a-028c8b0fb421" containerName="maas-api" Sep 14 10:00:01.338393 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.338367 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4d7cba-9c8e-4710-b562-1333b413e65c" containerName="payload-pre-processing" Sep 14 10:00:01.338393 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.338372 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4d7cba-9c8e-4710-b562-1333b413e65c" containerName="payload-pre-processing" Sep 14 10:00:01.338482 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.338434 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4d7cba-9c8e-4710-b562-1333b413e65c" containerName="payload-pre-processing" Sep 14 10:00:01.338482 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.338444 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7cda49c-5891-496b-8f2a-028c8b0fb421" containerName="maas-api" Sep 14 10:00:01.342915 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.342897 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr" Sep 14 10:00:01.346120 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.346100 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\"" Sep 14 10:00:01.354304 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.354283 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr"] Sep 14 10:00:01.464859 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.464817 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696ca118-4cc6-4f4c-8190-9c235d511f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8c98aa-gcttr\" (UID: \"696ca118-4cc6-4f4c-8190-9c235d511f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr" Sep 14 10:00:01.469527 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.469505 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk"] Sep 14 10:00:01.469701 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:00:01.469676 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95db1855_1a94_47a1_a5a6_856b8a49ff5c.slice/crio-6d75fe17fb910af86d6e28f94cfeba1e275131d49d045bcd9d8ff69c5cf4db33 WatchSource:0}: Error finding container 6d75fe17fb910af86d6e28f94cfeba1e275131d49d045bcd9d8ff69c5cf4db33: Status 404 returned error can't find the container with id 6d75fe17fb910af86d6e28f94cfeba1e275131d49d045bcd9d8ff69c5cf4db33 Sep 14 10:00:01.471352 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.471338 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 10:00:01.565459 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.565426 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696ca118-4cc6-4f4c-8190-9c235d511f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8c98aa-gcttr\" (UID: \"696ca118-4cc6-4f4c-8190-9c235d511f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr" Sep 14 10:00:01.566104 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.566084 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696ca118-4cc6-4f4c-8190-9c235d511f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8c98aa-gcttr\" (UID: \"696ca118-4cc6-4f4c-8190-9c235d511f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr" Sep 14 10:00:01.653570 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.653499 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr" Sep 14 10:00:01.661531 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.661481 2550 generic.go:356] "Generic (PLEG): container finished" podID="2f4d7cba-9c8e-4710-b562-1333b413e65c" containerID="e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084" exitCode=0 Sep 14 10:00:01.661693 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.661595 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" Sep 14 10:00:01.661693 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.661592 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" event={"ID":"2f4d7cba-9c8e-4710-b562-1333b413e65c","Type":"ContainerDied","Data":"e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084"} Sep 14 10:00:01.661822 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.661717 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx" event={"ID":"2f4d7cba-9c8e-4710-b562-1333b413e65c","Type":"ContainerDied","Data":"fd14fb644515538faf25757350e61154e704bd50c4ab730c4c11992c85035d9d"} Sep 14 10:00:01.661822 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.661774 2550 scope.go:117] "RemoveContainer" containerID="e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084" Sep 14 10:00:01.664388 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.663573 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" Sep 14 10:00:01.664388 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.663881 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9" event={"ID":"b7cda49c-5891-496b-8f2a-028c8b0fb421","Type":"ContainerDied","Data":"6d08286b3a953c8f0fd272990f833b9f2ea65e3a4ed5d999cbfe793dca29f056"} Sep 14 10:00:01.666045 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.665965 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk" event={"ID":"95db1855-1a94-47a1-a5a6-856b8a49ff5c","Type":"ContainerStarted","Data":"6d75fe17fb910af86d6e28f94cfeba1e275131d49d045bcd9d8ff69c5cf4db33"} Sep 14 10:00:01.687512 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.687409 2550 scope.go:117] "RemoveContainer" containerID="e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084" Sep 14 10:00:01.688089 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:00:01.688056 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084\": container with ID starting with e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084 not found: ID does not exist" containerID="e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084" Sep 14 10:00:01.688228 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.688100 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084"} err="failed to get container status \"e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084\": rpc error: code = NotFound desc = could not find container \"e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084\": container with ID starting with e250676f0942b34b285f48d4b7cc668e30c49d6c022aea6772872a94f0d07084 not found: ID does not exist" Sep 14 10:00:01.688228 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.688125 2550 scope.go:117] "RemoveContainer" containerID="1617110ca041c12a6101e8682dd783ada92c95d4c7c304891423f49f1458ef0a" Sep 14 10:00:01.716078 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.715113 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx"] Sep 14 10:00:01.721618 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.721466 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80fa7537-5f4cd4c578-8qcpx"] Sep 14 10:00:01.756769 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.756740 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9"] Sep 14 10:00:01.762870 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.762844 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-80fa7537-5949db8574-jtff9"] Sep 14 10:00:01.824957 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.824929 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f4d7cba-9c8e-4710-b562-1333b413e65c" path="/var/lib/kubelet/pods/2f4d7cba-9c8e-4710-b562-1333b413e65c/volumes" Sep 14 10:00:01.825442 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.825423 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7cda49c-5891-496b-8f2a-028c8b0fb421" path="/var/lib/kubelet/pods/b7cda49c-5891-496b-8f2a-028c8b0fb421/volumes" Sep 14 10:00:01.945762 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:01.945741 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr"] Sep 14 10:00:01.947609 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:00:01.947583 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696ca118_4cc6_4f4c_8190_9c235d511f68.slice/crio-bcaa55d930296eab6e09c4395c2873c21930fcb0626ee5a22942cb1016c2a4b7 WatchSource:0}: Error finding container bcaa55d930296eab6e09c4395c2873c21930fcb0626ee5a22942cb1016c2a4b7: Status 404 returned error can't find the container with id bcaa55d930296eab6e09c4395c2873c21930fcb0626ee5a22942cb1016c2a4b7 Sep 14 10:00:02.674373 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:02.674336 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr" event={"ID":"696ca118-4cc6-4f4c-8190-9c235d511f68","Type":"ContainerStarted","Data":"bcaa55d930296eab6e09c4395c2873c21930fcb0626ee5a22942cb1016c2a4b7"} Sep 14 10:00:05.693700 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:05.693619 2550 generic.go:356] "Generic (PLEG): container finished" podID="95db1855-1a94-47a1-a5a6-856b8a49ff5c" containerID="0b34bb31eaed0d035f1fe30d269a1ebce6136f719f8fda82917979307f278c17" exitCode=0 Sep 14 10:00:05.693700 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:05.693676 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk" event={"ID":"95db1855-1a94-47a1-a5a6-856b8a49ff5c","Type":"ContainerDied","Data":"0b34bb31eaed0d035f1fe30d269a1ebce6136f719f8fda82917979307f278c17"} Sep 14 10:00:05.695307 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:05.695286 2550 generic.go:356] "Generic (PLEG): container finished" podID="696ca118-4cc6-4f4c-8190-9c235d511f68" containerID="51c241cb3956c9be31c4cda18d0f47232fa4e9d5a2d9368e05ab7bfaf6afabab" exitCode=0 Sep 14 10:00:05.695389 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:05.695323 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr" event={"ID":"696ca118-4cc6-4f4c-8190-9c235d511f68","Type":"ContainerDied","Data":"51c241cb3956c9be31c4cda18d0f47232fa4e9d5a2d9368e05ab7bfaf6afabab"} Sep 14 10:00:06.837604 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:06.837580 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk" Sep 14 10:00:06.876675 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:06.876646 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr" Sep 14 10:00:06.915273 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:06.915237 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696ca118-4cc6-4f4c-8190-9c235d511f68-maas-api-service-ca\") pod \"696ca118-4cc6-4f4c-8190-9c235d511f68\" (UID: \"696ca118-4cc6-4f4c-8190-9c235d511f68\") " Sep 14 10:00:06.915567 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:06.915544 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696ca118-4cc6-4f4c-8190-9c235d511f68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "696ca118-4cc6-4f4c-8190-9c235d511f68" (UID: "696ca118-4cc6-4f4c-8190-9c235d511f68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:00:06.915693 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:06.915673 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696ca118-4cc6-4f4c-8190-9c235d511f68-maas-api-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:00:07.707914 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:07.707885 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk" Sep 14 10:00:07.708276 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:07.707888 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk" event={"ID":"95db1855-1a94-47a1-a5a6-856b8a49ff5c","Type":"ContainerDied","Data":"6d75fe17fb910af86d6e28f94cfeba1e275131d49d045bcd9d8ff69c5cf4db33"} Sep 14 10:00:07.708276 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:07.708018 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d75fe17fb910af86d6e28f94cfeba1e275131d49d045bcd9d8ff69c5cf4db33" Sep 14 10:00:07.709256 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:07.709238 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr" Sep 14 10:00:07.709392 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:07.709282 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr" event={"ID":"696ca118-4cc6-4f4c-8190-9c235d511f68","Type":"ContainerDied","Data":"bcaa55d930296eab6e09c4395c2873c21930fcb0626ee5a22942cb1016c2a4b7"} Sep 14 10:00:07.709392 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:07.709327 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcaa55d930296eab6e09c4395c2873c21930fcb0626ee5a22942cb1016c2a4b7" Sep 14 10:00:12.272986 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.272953 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg"] Sep 14 10:00:12.273440 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.273197 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" podUID="7879e468-9b31-49dc-a21a-e5850db2365d" containerName="payload-pre-processing" containerID="cri-o://fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb" gracePeriod=30 Sep 14 10:00:12.500802 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.500769 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr"] Sep 14 10:00:12.502894 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.502872 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8c98aa-gcttr"] Sep 14 10:00:12.536702 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.536622 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 10:00:12.669680 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.669651 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4zhn\" (UniqueName: \"kubernetes.io/projected/7879e468-9b31-49dc-a21a-e5850db2365d-kube-api-access-v4zhn\") pod \"7879e468-9b31-49dc-a21a-e5850db2365d\" (UID: \"7879e468-9b31-49dc-a21a-e5850db2365d\") " Sep 14 10:00:12.669861 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.669699 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7879e468-9b31-49dc-a21a-e5850db2365d-plugins-config-volume\") pod \"7879e468-9b31-49dc-a21a-e5850db2365d\" (UID: \"7879e468-9b31-49dc-a21a-e5850db2365d\") " Sep 14 10:00:12.670108 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.670084 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7879e468-9b31-49dc-a21a-e5850db2365d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7879e468-9b31-49dc-a21a-e5850db2365d" (UID: "7879e468-9b31-49dc-a21a-e5850db2365d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:00:12.671713 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.671692 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7879e468-9b31-49dc-a21a-e5850db2365d-kube-api-access-v4zhn" (OuterVolumeSpecName: "kube-api-access-v4zhn") pod "7879e468-9b31-49dc-a21a-e5850db2365d" (UID: "7879e468-9b31-49dc-a21a-e5850db2365d"). InnerVolumeSpecName "kube-api-access-v4zhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:00:12.737928 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.737892 2550 generic.go:356] "Generic (PLEG): container finished" podID="7879e468-9b31-49dc-a21a-e5850db2365d" containerID="fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb" exitCode=0 Sep 14 10:00:12.738112 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.737941 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" Sep 14 10:00:12.738112 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.737980 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" event={"ID":"7879e468-9b31-49dc-a21a-e5850db2365d","Type":"ContainerDied","Data":"fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb"} Sep 14 10:00:12.738112 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.738036 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg" event={"ID":"7879e468-9b31-49dc-a21a-e5850db2365d","Type":"ContainerDied","Data":"6044b75545e5c100a7cc712ae997d03eb8d6507b5ef60b0bfd0e4ceccaf9f5f5"} Sep 14 10:00:12.738112 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.738062 2550 scope.go:117] "RemoveContainer" containerID="fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb" Sep 14 10:00:12.749944 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.749924 2550 scope.go:117] "RemoveContainer" containerID="fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb" Sep 14 10:00:12.750266 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:00:12.750242 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb\": container with ID starting with fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb not found: ID does not exist" containerID="fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb" Sep 14 10:00:12.750339 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.750271 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb"} err="failed to get container status \"fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb\": rpc error: code = NotFound desc = could not find container \"fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb\": container with ID starting with fcfd9d08139e53b04072842f204bcf8de0e85fbbd6abf83556af2bf04c1a2aeb not found: ID does not exist" Sep 14 10:00:12.770495 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.770431 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zhn\" (UniqueName: \"kubernetes.io/projected/7879e468-9b31-49dc-a21a-e5850db2365d-kube-api-access-v4zhn\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:00:12.770495 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.770462 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7879e468-9b31-49dc-a21a-e5850db2365d-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:00:12.772327 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.772305 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg"] Sep 14 10:00:12.779976 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:12.779955 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8c98aa-58c544d96b-jwqwg"] Sep 14 10:00:13.825621 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:13.825582 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="696ca118-4cc6-4f4c-8190-9c235d511f68" path="/var/lib/kubelet/pods/696ca118-4cc6-4f4c-8190-9c235d511f68/volumes" Sep 14 10:00:13.826158 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:00:13.826138 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7879e468-9b31-49dc-a21a-e5850db2365d" path="/var/lib/kubelet/pods/7879e468-9b31-49dc-a21a-e5850db2365d/volumes" Sep 14 10:01:07.188723 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.188688 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq"] Sep 14 10:01:07.189195 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.189180 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95db1855-1a94-47a1-a5a6-856b8a49ff5c" containerName="cleanup" Sep 14 10:01:07.189239 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.189196 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="95db1855-1a94-47a1-a5a6-856b8a49ff5c" containerName="cleanup" Sep 14 10:01:07.189239 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.189217 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696ca118-4cc6-4f4c-8190-9c235d511f68" containerName="revoke-keys" Sep 14 10:01:07.189239 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.189226 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="696ca118-4cc6-4f4c-8190-9c235d511f68" containerName="revoke-keys" Sep 14 10:01:07.189335 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.189249 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7879e468-9b31-49dc-a21a-e5850db2365d" containerName="payload-pre-processing" Sep 14 10:01:07.189335 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.189254 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7879e468-9b31-49dc-a21a-e5850db2365d" containerName="payload-pre-processing" Sep 14 10:01:07.189335 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.189324 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="95db1855-1a94-47a1-a5a6-856b8a49ff5c" containerName="cleanup" Sep 14 10:01:07.189335 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.189335 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7879e468-9b31-49dc-a21a-e5850db2365d" containerName="payload-pre-processing" Sep 14 10:01:07.189454 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.189344 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="696ca118-4cc6-4f4c-8190-9c235d511f68" containerName="revoke-keys" Sep 14 10:01:07.193933 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.193916 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:01:07.196489 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.196464 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-31f3b6\"" Sep 14 10:01:07.196619 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.196503 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-31f3b6-dockercfg-zlnpr\"" Sep 14 10:01:07.207288 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.207260 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq"] Sep 14 10:01:07.291260 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.291225 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxgp\" (UniqueName: \"kubernetes.io/projected/fd291302-1abf-4948-b740-19f71dfa957e-kube-api-access-zdxgp\") pod \"payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq\" (UID: \"fd291302-1abf-4948-b740-19f71dfa957e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:01:07.291412 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.291276 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd291302-1abf-4948-b740-19f71dfa957e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq\" (UID: \"fd291302-1abf-4948-b740-19f71dfa957e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:01:07.392710 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.392676 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxgp\" (UniqueName: \"kubernetes.io/projected/fd291302-1abf-4948-b740-19f71dfa957e-kube-api-access-zdxgp\") pod \"payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq\" (UID: \"fd291302-1abf-4948-b740-19f71dfa957e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:01:07.392865 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.392723 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd291302-1abf-4948-b740-19f71dfa957e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq\" (UID: \"fd291302-1abf-4948-b740-19f71dfa957e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:01:07.393433 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.393414 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd291302-1abf-4948-b740-19f71dfa957e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq\" (UID: \"fd291302-1abf-4948-b740-19f71dfa957e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:01:07.401043 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.401021 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxgp\" (UniqueName: \"kubernetes.io/projected/fd291302-1abf-4948-b740-19f71dfa957e-kube-api-access-zdxgp\") pod \"payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq\" (UID: \"fd291302-1abf-4948-b740-19f71dfa957e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:01:07.513409 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.513331 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:01:07.789452 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:07.789428 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq"] Sep 14 10:01:07.791218 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:01:07.791192 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd291302_1abf_4948_b740_19f71dfa957e.slice/crio-e823e07557b0a6704f5923392c36994f06a1071fc277e32a8a54716dfef567f2 WatchSource:0}: Error finding container e823e07557b0a6704f5923392c36994f06a1071fc277e32a8a54716dfef567f2: Status 404 returned error can't find the container with id e823e07557b0a6704f5923392c36994f06a1071fc277e32a8a54716dfef567f2 Sep 14 10:01:08.039489 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:08.039402 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" event={"ID":"fd291302-1abf-4948-b740-19f71dfa957e","Type":"ContainerStarted","Data":"8ba99fd89698ce728501ec5f79a595311efc16a50c5b8db64ceb1b40b175d19e"} Sep 14 10:01:08.039489 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:08.039459 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" event={"ID":"fd291302-1abf-4948-b740-19f71dfa957e","Type":"ContainerStarted","Data":"e823e07557b0a6704f5923392c36994f06a1071fc277e32a8a54716dfef567f2"} Sep 14 10:01:08.039489 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:08.039481 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:01:08.062424 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:08.062379 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" podStartSLOduration=1.062365502 podStartE2EDuration="1.062365502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:01:08.060922072 +0000 UTC m=+1084.756047392" watchObservedRunningTime="2026-09-14 10:01:08.062365502 +0000 UTC m=+1084.757490807" Sep 14 10:01:19.046268 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:19.046236 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:01:55.883725 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:55.883687 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk"] Sep 14 10:01:55.886957 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:55.886938 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:55.892392 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:55.892372 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bc008e9d-serving-cert\"" Sep 14 10:01:55.912541 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:55.912515 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk"] Sep 14 10:01:55.960349 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:55.960314 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-openshift-service-ca\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:55.960349 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:55.960352 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhqth\" (UniqueName: \"kubernetes.io/projected/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-kube-api-access-nhqth\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:55.960558 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:55.960470 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-maas-api-tls\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:55.960558 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:55.960532 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-metrics-certs\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:56.061865 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.061823 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-maas-api-tls\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:56.062059 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.061879 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-metrics-certs\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:56.062059 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.061944 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-openshift-service-ca\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:56.062059 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.061971 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhqth\" (UniqueName: \"kubernetes.io/projected/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-kube-api-access-nhqth\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:56.062683 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.062659 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-openshift-service-ca\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:56.064527 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.064501 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-metrics-certs\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:56.064677 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.064546 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-maas-api-tls\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:56.073510 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.073486 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhqth\" (UniqueName: \"kubernetes.io/projected/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-kube-api-access-nhqth\") pod \"maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:56.198169 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.198141 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:56.480225 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.480200 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk"] Sep 14 10:01:56.481817 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:01:56.481785 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c82ae7_a59d_45a9_a18c_3c8291b0bf0d.slice/crio-931ffb0ed14379ce7f5f06febd1aff8671cff1176e750c0f3b72cd6235f66471 WatchSource:0}: Error finding container 931ffb0ed14379ce7f5f06febd1aff8671cff1176e750c0f3b72cd6235f66471: Status 404 returned error can't find the container with id 931ffb0ed14379ce7f5f06febd1aff8671cff1176e750c0f3b72cd6235f66471 Sep 14 10:01:56.701078 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.701049 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh"] Sep 14 10:01:56.704728 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.704712 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:01:56.709283 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.709260 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bc008e9d-dockercfg-tbk92\"" Sep 14 10:01:56.709808 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.709794 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bc008e9d\"" Sep 14 10:01:56.732716 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.732660 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh"] Sep 14 10:01:56.768685 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.768649 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8066bb1b-62bd-40f3-970f-b735e2fa9150-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh\" (UID: \"8066bb1b-62bd-40f3-970f-b735e2fa9150\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:01:56.768855 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.768722 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwp5\" (UniqueName: \"kubernetes.io/projected/8066bb1b-62bd-40f3-970f-b735e2fa9150-kube-api-access-2fwp5\") pod \"payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh\" (UID: \"8066bb1b-62bd-40f3-970f-b735e2fa9150\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:01:56.799443 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.799412 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w"] Sep 14 10:01:56.802369 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.802351 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:01:56.822908 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.822883 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w"] Sep 14 10:01:56.869902 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.869874 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8066bb1b-62bd-40f3-970f-b735e2fa9150-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh\" (UID: \"8066bb1b-62bd-40f3-970f-b735e2fa9150\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:01:56.870052 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.869921 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwp5\" (UniqueName: \"kubernetes.io/projected/8066bb1b-62bd-40f3-970f-b735e2fa9150-kube-api-access-2fwp5\") pod \"payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh\" (UID: \"8066bb1b-62bd-40f3-970f-b735e2fa9150\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:01:56.870052 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.869953 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgfgr\" (UniqueName: \"kubernetes.io/projected/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-kube-api-access-fgfgr\") pod \"payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w\" (UID: \"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf\") " pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:01:56.870052 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.870022 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-plugins-config-volume\") pod \"payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w\" (UID: \"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf\") " pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:01:56.870549 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.870530 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8066bb1b-62bd-40f3-970f-b735e2fa9150-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh\" (UID: \"8066bb1b-62bd-40f3-970f-b735e2fa9150\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:01:56.880110 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.880083 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwp5\" (UniqueName: \"kubernetes.io/projected/8066bb1b-62bd-40f3-970f-b735e2fa9150-kube-api-access-2fwp5\") pod \"payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh\" (UID: \"8066bb1b-62bd-40f3-970f-b735e2fa9150\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:01:56.971213 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.971186 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgfgr\" (UniqueName: \"kubernetes.io/projected/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-kube-api-access-fgfgr\") pod \"payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w\" (UID: \"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf\") " pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:01:56.971572 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.971287 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-plugins-config-volume\") pod \"payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w\" (UID: \"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf\") " pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:01:56.971963 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.971945 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-plugins-config-volume\") pod \"payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w\" (UID: \"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf\") " pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:01:56.980610 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:56.980589 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgfgr\" (UniqueName: \"kubernetes.io/projected/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-kube-api-access-fgfgr\") pod \"payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w\" (UID: \"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf\") " pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:01:57.015497 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:57.015434 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:01:57.115112 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:57.115071 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:01:57.295915 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:57.295884 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh"] Sep 14 10:01:57.296915 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:01:57.296891 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8066bb1b_62bd_40f3_970f_b735e2fa9150.slice/crio-a0d483c471139d3fd0e730e31485345fed148fdb108004c72e2fa2d16c068a93 WatchSource:0}: Error finding container a0d483c471139d3fd0e730e31485345fed148fdb108004c72e2fa2d16c068a93: Status 404 returned error can't find the container with id a0d483c471139d3fd0e730e31485345fed148fdb108004c72e2fa2d16c068a93 Sep 14 10:01:57.310784 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:57.310752 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" event={"ID":"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d","Type":"ContainerStarted","Data":"58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713"} Sep 14 10:01:57.310896 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:57.310814 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:01:57.310896 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:57.310840 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" event={"ID":"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d","Type":"ContainerStarted","Data":"931ffb0ed14379ce7f5f06febd1aff8671cff1176e750c0f3b72cd6235f66471"} Sep 14 10:01:57.311993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:57.311962 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" event={"ID":"8066bb1b-62bd-40f3-970f-b735e2fa9150","Type":"ContainerStarted","Data":"a0d483c471139d3fd0e730e31485345fed148fdb108004c72e2fa2d16c068a93"} Sep 14 10:01:57.331623 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:57.331241 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" podStartSLOduration=1.927237988 podStartE2EDuration="2.33121031s" totalImagesPullingTime="403.972322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:01:57.331083786 +0000 UTC m=+1134.026209103" watchObservedRunningTime="2026-09-14 10:01:57.33121031 +0000 UTC m=+1134.026335614" Sep 14 10:01:57.395499 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:57.395474 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w"] Sep 14 10:01:57.397933 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:01:57.397907 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd4b5f2_17d2_4f57_a89b_b20e1c58a8bf.slice/crio-51746270e452d6137f3d03fa9e9ee40391e1ab0236324a5ecb4eb40ab71735dc WatchSource:0}: Error finding container 51746270e452d6137f3d03fa9e9ee40391e1ab0236324a5ecb4eb40ab71735dc: Status 404 returned error can't find the container with id 51746270e452d6137f3d03fa9e9ee40391e1ab0236324a5ecb4eb40ab71735dc Sep 14 10:01:58.318494 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:58.318456 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" event={"ID":"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf","Type":"ContainerStarted","Data":"aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087"} Sep 14 10:01:58.318984 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:58.318518 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" event={"ID":"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf","Type":"ContainerStarted","Data":"51746270e452d6137f3d03fa9e9ee40391e1ab0236324a5ecb4eb40ab71735dc"} Sep 14 10:01:58.318984 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:58.318548 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:01:58.320133 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:58.320105 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" event={"ID":"8066bb1b-62bd-40f3-970f-b735e2fa9150","Type":"ContainerStarted","Data":"e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833"} Sep 14 10:01:58.320254 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:58.320187 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:01:58.342239 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:58.342184 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" podStartSLOduration=2.342164222 podStartE2EDuration="2.342164222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:01:58.337999893 +0000 UTC m=+1135.033125210" watchObservedRunningTime="2026-09-14 10:01:58.342164222 +0000 UTC m=+1135.037289528" Sep 14 10:01:58.362159 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:01:58.362118 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" podStartSLOduration=2.362105061 podStartE2EDuration="2.362105061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:01:58.360751843 +0000 UTC m=+1135.055877150" watchObservedRunningTime="2026-09-14 10:01:58.362105061 +0000 UTC m=+1135.057230402" Sep 14 10:02:03.326823 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:03.326790 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:02:09.328406 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:09.328373 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:02:09.328806 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:09.328441 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:02:38.551590 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.551555 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk"] Sep 14 10:02:38.552375 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.551773 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" podUID="79c82ae7-a59d-45a9-a18c-3c8291b0bf0d" containerName="maas-api" containerID="cri-o://58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713" gracePeriod=30 Sep 14 10:02:38.584210 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.584162 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w"] Sep 14 10:02:38.585788 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.584868 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" podUID="8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf" containerName="payload-processing" containerID="cri-o://aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087" gracePeriod=30 Sep 14 10:02:38.607030 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.606992 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh"] Sep 14 10:02:38.607395 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.607360 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" podUID="8066bb1b-62bd-40f3-970f-b735e2fa9150" containerName="payload-pre-processing" containerID="cri-o://e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833" gracePeriod=30 Sep 14 10:02:38.885316 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.885290 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:02:38.908359 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.908321 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:02:38.923584 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.923559 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:02:38.982824 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.982795 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-metrics-certs\") pod \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " Sep 14 10:02:38.983016 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.982832 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-openshift-service-ca\") pod \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " Sep 14 10:02:38.983016 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.982858 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgfgr\" (UniqueName: \"kubernetes.io/projected/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-kube-api-access-fgfgr\") pod \"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf\" (UID: \"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf\") " Sep 14 10:02:38.983016 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.982885 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fwp5\" (UniqueName: \"kubernetes.io/projected/8066bb1b-62bd-40f3-970f-b735e2fa9150-kube-api-access-2fwp5\") pod \"8066bb1b-62bd-40f3-970f-b735e2fa9150\" (UID: \"8066bb1b-62bd-40f3-970f-b735e2fa9150\") " Sep 14 10:02:38.983016 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.983000 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-plugins-config-volume\") pod \"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf\" (UID: \"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf\") " Sep 14 10:02:38.983247 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.983040 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8066bb1b-62bd-40f3-970f-b735e2fa9150-plugins-config-volume\") pod \"8066bb1b-62bd-40f3-970f-b735e2fa9150\" (UID: \"8066bb1b-62bd-40f3-970f-b735e2fa9150\") " Sep 14 10:02:38.983247 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.983083 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhqth\" (UniqueName: \"kubernetes.io/projected/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-kube-api-access-nhqth\") pod \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " Sep 14 10:02:38.983247 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.983162 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-maas-api-tls\") pod \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\" (UID: \"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d\") " Sep 14 10:02:38.983467 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.983259 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79c82ae7-a59d-45a9-a18c-3c8291b0bf0d" (UID: "79c82ae7-a59d-45a9-a18c-3c8291b0bf0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:02:38.983467 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.983450 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8066bb1b-62bd-40f3-970f-b735e2fa9150-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8066bb1b-62bd-40f3-970f-b735e2fa9150" (UID: "8066bb1b-62bd-40f3-970f-b735e2fa9150"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:02:38.983629 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.983611 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8066bb1b-62bd-40f3-970f-b735e2fa9150-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:02:38.983712 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.983627 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf" (UID: "8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:02:38.983712 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.983652 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:02:38.985176 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.985151 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "79c82ae7-a59d-45a9-a18c-3c8291b0bf0d" (UID: "79c82ae7-a59d-45a9-a18c-3c8291b0bf0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:02:38.985289 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.985251 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-kube-api-access-nhqth" (OuterVolumeSpecName: "kube-api-access-nhqth") pod "79c82ae7-a59d-45a9-a18c-3c8291b0bf0d" (UID: "79c82ae7-a59d-45a9-a18c-3c8291b0bf0d"). InnerVolumeSpecName "kube-api-access-nhqth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:02:38.985289 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.985281 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-kube-api-access-fgfgr" (OuterVolumeSpecName: "kube-api-access-fgfgr") pod "8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf" (UID: "8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf"). InnerVolumeSpecName "kube-api-access-fgfgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:02:38.985473 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.985453 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79c82ae7-a59d-45a9-a18c-3c8291b0bf0d" (UID: "79c82ae7-a59d-45a9-a18c-3c8291b0bf0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:02:38.985473 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:38.985464 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8066bb1b-62bd-40f3-970f-b735e2fa9150-kube-api-access-2fwp5" (OuterVolumeSpecName: "kube-api-access-2fwp5") pod "8066bb1b-62bd-40f3-970f-b735e2fa9150" (UID: "8066bb1b-62bd-40f3-970f-b735e2fa9150"). InnerVolumeSpecName "kube-api-access-2fwp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:02:39.084456 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.084383 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:02:39.084456 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.084412 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgfgr\" (UniqueName: \"kubernetes.io/projected/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-kube-api-access-fgfgr\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:02:39.084456 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.084427 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fwp5\" (UniqueName: \"kubernetes.io/projected/8066bb1b-62bd-40f3-970f-b735e2fa9150-kube-api-access-2fwp5\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:02:39.084456 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.084440 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:02:39.084456 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.084452 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhqth\" (UniqueName: \"kubernetes.io/projected/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-kube-api-access-nhqth\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:02:39.084743 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.084464 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:02:39.544307 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.544273 2550 generic.go:356] "Generic (PLEG): container finished" podID="79c82ae7-a59d-45a9-a18c-3c8291b0bf0d" containerID="58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713" exitCode=0 Sep 14 10:02:39.544503 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.544316 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" Sep 14 10:02:39.544503 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.544360 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" event={"ID":"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d","Type":"ContainerDied","Data":"58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713"} Sep 14 10:02:39.544503 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.544419 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk" event={"ID":"79c82ae7-a59d-45a9-a18c-3c8291b0bf0d","Type":"ContainerDied","Data":"931ffb0ed14379ce7f5f06febd1aff8671cff1176e750c0f3b72cd6235f66471"} Sep 14 10:02:39.544503 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.544445 2550 scope.go:117] "RemoveContainer" containerID="58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713" Sep 14 10:02:39.545801 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.545782 2550 generic.go:356] "Generic (PLEG): container finished" podID="8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf" containerID="aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087" exitCode=0 Sep 14 10:02:39.545878 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.545836 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" Sep 14 10:02:39.545934 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.545891 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" event={"ID":"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf","Type":"ContainerDied","Data":"aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087"} Sep 14 10:02:39.545980 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.545929 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w" event={"ID":"8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf","Type":"ContainerDied","Data":"51746270e452d6137f3d03fa9e9ee40391e1ab0236324a5ecb4eb40ab71735dc"} Sep 14 10:02:39.547231 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.547205 2550 generic.go:356] "Generic (PLEG): container finished" podID="8066bb1b-62bd-40f3-970f-b735e2fa9150" containerID="e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833" exitCode=0 Sep 14 10:02:39.547331 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.547251 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" Sep 14 10:02:39.547331 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.547297 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" event={"ID":"8066bb1b-62bd-40f3-970f-b735e2fa9150","Type":"ContainerDied","Data":"e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833"} Sep 14 10:02:39.547437 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.547346 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh" event={"ID":"8066bb1b-62bd-40f3-970f-b735e2fa9150","Type":"ContainerDied","Data":"a0d483c471139d3fd0e730e31485345fed148fdb108004c72e2fa2d16c068a93"} Sep 14 10:02:39.554738 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.554537 2550 scope.go:117] "RemoveContainer" containerID="58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713" Sep 14 10:02:39.555041 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:02:39.554839 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713\": container with ID starting with 58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713 not found: ID does not exist" containerID="58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713" Sep 14 10:02:39.555041 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.554874 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713"} err="failed to get container status \"58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713\": rpc error: code = NotFound desc = could not find container \"58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713\": container with ID starting with 58fcc5d0ca0f4400dfe32f537f05c1ce45f7afbf5017a5e3f0d4cd947df9e713 not found: ID does not exist" Sep 14 10:02:39.555041 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.554897 2550 scope.go:117] "RemoveContainer" containerID="aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087" Sep 14 10:02:39.579150 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.579130 2550 scope.go:117] "RemoveContainer" containerID="aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087" Sep 14 10:02:39.579425 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:02:39.579402 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087\": container with ID starting with aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087 not found: ID does not exist" containerID="aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087" Sep 14 10:02:39.579501 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.579430 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087"} err="failed to get container status \"aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087\": rpc error: code = NotFound desc = could not find container \"aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087\": container with ID starting with aa019b83554d2a95de63ecdc226fd9abb9e252f8be484433b27a6381b78de087 not found: ID does not exist" Sep 14 10:02:39.579501 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.579445 2550 scope.go:117] "RemoveContainer" containerID="e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833" Sep 14 10:02:39.587297 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.587255 2550 scope.go:117] "RemoveContainer" containerID="e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833" Sep 14 10:02:39.587806 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:02:39.587767 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833\": container with ID starting with e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833 not found: ID does not exist" containerID="e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833" Sep 14 10:02:39.587883 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.587814 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833"} err="failed to get container status \"e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833\": rpc error: code = NotFound desc = could not find container \"e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833\": container with ID starting with e9efefbe64ac7ffaee8afa03f37dde4d644c7d8d964e7cd9ebb4047ebc116833 not found: ID does not exist" Sep 14 10:02:39.589809 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.589790 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk"] Sep 14 10:02:39.596084 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.596065 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc008e9d-695c9d579d-28hbk"] Sep 14 10:02:39.606598 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.606579 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh"] Sep 14 10:02:39.610514 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.610495 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc008e9d-76b8587ff4-kxkzh"] Sep 14 10:02:39.621235 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.621208 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w"] Sep 14 10:02:39.626682 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.626661 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc008e9d-5c9798657-tqv2w"] Sep 14 10:02:39.825143 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.825053 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c82ae7-a59d-45a9-a18c-3c8291b0bf0d" path="/var/lib/kubelet/pods/79c82ae7-a59d-45a9-a18c-3c8291b0bf0d/volumes" Sep 14 10:02:39.825505 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.825484 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8066bb1b-62bd-40f3-970f-b735e2fa9150" path="/var/lib/kubelet/pods/8066bb1b-62bd-40f3-970f-b735e2fa9150/volumes" Sep 14 10:02:39.825902 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:39.825882 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf" path="/var/lib/kubelet/pods/8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf/volumes" Sep 14 10:02:42.428472 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.428437 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq"] Sep 14 10:02:42.428956 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.428679 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" podUID="fd291302-1abf-4948-b740-19f71dfa957e" containerName="payload-pre-processing" containerID="cri-o://8ba99fd89698ce728501ec5f79a595311efc16a50c5b8db64ceb1b40b175d19e" gracePeriod=30 Sep 14 10:02:42.570906 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.570871 2550 generic.go:356] "Generic (PLEG): container finished" podID="fd291302-1abf-4948-b740-19f71dfa957e" containerID="8ba99fd89698ce728501ec5f79a595311efc16a50c5b8db64ceb1b40b175d19e" exitCode=0 Sep 14 10:02:42.571063 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.570935 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" event={"ID":"fd291302-1abf-4948-b740-19f71dfa957e","Type":"ContainerDied","Data":"8ba99fd89698ce728501ec5f79a595311efc16a50c5b8db64ceb1b40b175d19e"} Sep 14 10:02:42.699101 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.699080 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:02:42.821578 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.821539 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdxgp\" (UniqueName: \"kubernetes.io/projected/fd291302-1abf-4948-b740-19f71dfa957e-kube-api-access-zdxgp\") pod \"fd291302-1abf-4948-b740-19f71dfa957e\" (UID: \"fd291302-1abf-4948-b740-19f71dfa957e\") " Sep 14 10:02:42.821779 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.821728 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd291302-1abf-4948-b740-19f71dfa957e-plugins-config-volume\") pod \"fd291302-1abf-4948-b740-19f71dfa957e\" (UID: \"fd291302-1abf-4948-b740-19f71dfa957e\") " Sep 14 10:02:42.822082 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.822053 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd291302-1abf-4948-b740-19f71dfa957e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd291302-1abf-4948-b740-19f71dfa957e" (UID: "fd291302-1abf-4948-b740-19f71dfa957e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:02:42.823744 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.823712 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd291302-1abf-4948-b740-19f71dfa957e-kube-api-access-zdxgp" (OuterVolumeSpecName: "kube-api-access-zdxgp") pod "fd291302-1abf-4948-b740-19f71dfa957e" (UID: "fd291302-1abf-4948-b740-19f71dfa957e"). InnerVolumeSpecName "kube-api-access-zdxgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:02:42.923043 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.922998 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd291302-1abf-4948-b740-19f71dfa957e-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:02:42.923043 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:42.923035 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxgp\" (UniqueName: \"kubernetes.io/projected/fd291302-1abf-4948-b740-19f71dfa957e-kube-api-access-zdxgp\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:02:43.577987 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:43.577955 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" Sep 14 10:02:43.577987 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:43.577978 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq" event={"ID":"fd291302-1abf-4948-b740-19f71dfa957e","Type":"ContainerDied","Data":"e823e07557b0a6704f5923392c36994f06a1071fc277e32a8a54716dfef567f2"} Sep 14 10:02:43.578496 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:43.578048 2550 scope.go:117] "RemoveContainer" containerID="8ba99fd89698ce728501ec5f79a595311efc16a50c5b8db64ceb1b40b175d19e" Sep 14 10:02:43.600307 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:43.600268 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq"] Sep 14 10:02:43.606731 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:43.606707 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-31f3b6-5bdf696478-mc9jq"] Sep 14 10:02:43.826895 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:02:43.826862 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd291302-1abf-4948-b740-19f71dfa957e" path="/var/lib/kubelet/pods/fd291302-1abf-4948-b740-19f71dfa957e/volumes" Sep 14 10:03:03.872665 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:03.872612 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:03:03.875571 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:03.875544 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:03:08.463187 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463154 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr"] Sep 14 10:03:08.463686 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463590 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf" containerName="payload-processing" Sep 14 10:03:08.463686 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463602 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf" containerName="payload-processing" Sep 14 10:03:08.463686 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463616 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd291302-1abf-4948-b740-19f71dfa957e" containerName="payload-pre-processing" Sep 14 10:03:08.463686 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463622 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd291302-1abf-4948-b740-19f71dfa957e" containerName="payload-pre-processing" Sep 14 10:03:08.463686 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463629 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c82ae7-a59d-45a9-a18c-3c8291b0bf0d" containerName="maas-api" Sep 14 10:03:08.463686 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463665 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c82ae7-a59d-45a9-a18c-3c8291b0bf0d" containerName="maas-api" Sep 14 10:03:08.463686 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463679 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8066bb1b-62bd-40f3-970f-b735e2fa9150" containerName="payload-pre-processing" Sep 14 10:03:08.463686 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463685 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8066bb1b-62bd-40f3-970f-b735e2fa9150" containerName="payload-pre-processing" Sep 14 10:03:08.464116 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463749 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c82ae7-a59d-45a9-a18c-3c8291b0bf0d" containerName="maas-api" Sep 14 10:03:08.464116 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463759 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd291302-1abf-4948-b740-19f71dfa957e" containerName="payload-pre-processing" Sep 14 10:03:08.464116 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463766 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd4b5f2-17d2-4f57-a89b-b20e1c58a8bf" containerName="payload-processing" Sep 14 10:03:08.464116 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.463773 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8066bb1b-62bd-40f3-970f-b735e2fa9150" containerName="payload-pre-processing" Sep 14 10:03:08.468281 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.468255 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.472368 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.472149 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-8fd006-serving-cert\"" Sep 14 10:03:08.479858 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.478429 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr"] Sep 14 10:03:08.569416 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.569383 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.569595 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.569439 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.569595 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.569519 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbj4h\" (UniqueName: \"kubernetes.io/projected/cf406175-8413-4320-9043-0188ffc52141-kube-api-access-kbj4h\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.569704 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.569658 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf406175-8413-4320-9043-0188ffc52141-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.670351 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.670272 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.670561 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.670421 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbj4h\" (UniqueName: \"kubernetes.io/projected/cf406175-8413-4320-9043-0188ffc52141-kube-api-access-kbj4h\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.670561 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.670543 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf406175-8413-4320-9043-0188ffc52141-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.670705 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.670571 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.671396 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.671368 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf406175-8413-4320-9043-0188ffc52141-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.672956 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.672933 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.673078 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.672983 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.679336 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.679311 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbj4h\" (UniqueName: \"kubernetes.io/projected/cf406175-8413-4320-9043-0188ffc52141-kube-api-access-kbj4h\") pod \"maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:08.787051 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:08.786982 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:09.063613 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.063580 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr"] Sep 14 10:03:09.064556 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:03:09.064527 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf406175_8413_4320_9043_0188ffc52141.slice/crio-e8007896ba77f22e48a8cf81eca8911151168d597f662cdaf57b9a02c55d1294 WatchSource:0}: Error finding container e8007896ba77f22e48a8cf81eca8911151168d597f662cdaf57b9a02c55d1294: Status 404 returned error can't find the container with id e8007896ba77f22e48a8cf81eca8911151168d597f662cdaf57b9a02c55d1294 Sep 14 10:03:09.141684 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.141649 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42"] Sep 14 10:03:09.146425 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.146404 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:03:09.149277 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.149253 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-8fd006\"" Sep 14 10:03:09.157289 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.157268 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42"] Sep 14 10:03:09.176302 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.176274 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baba4bae-13e2-43e7-a5af-ad81a475d96b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42\" (UID: \"baba4bae-13e2-43e7-a5af-ad81a475d96b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:03:09.176410 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.176316 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8654\" (UniqueName: \"kubernetes.io/projected/baba4bae-13e2-43e7-a5af-ad81a475d96b-kube-api-access-g8654\") pod \"payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42\" (UID: \"baba4bae-13e2-43e7-a5af-ad81a475d96b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:03:09.213454 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.213420 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv"] Sep 14 10:03:09.217033 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.217016 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:03:09.220597 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.220575 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-8fd006-dockercfg-9qbmk\"" Sep 14 10:03:09.229689 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.229668 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv"] Sep 14 10:03:09.277078 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.277045 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baba4bae-13e2-43e7-a5af-ad81a475d96b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42\" (UID: \"baba4bae-13e2-43e7-a5af-ad81a475d96b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:03:09.277251 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.277092 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ace602e-4cae-4e21-8b8c-efff41247ee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv\" (UID: \"1ace602e-4cae-4e21-8b8c-efff41247ee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:03:09.277251 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.277121 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkk5\" (UniqueName: \"kubernetes.io/projected/1ace602e-4cae-4e21-8b8c-efff41247ee7-kube-api-access-dwkk5\") pod \"payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv\" (UID: \"1ace602e-4cae-4e21-8b8c-efff41247ee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:03:09.277251 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.277154 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8654\" (UniqueName: \"kubernetes.io/projected/baba4bae-13e2-43e7-a5af-ad81a475d96b-kube-api-access-g8654\") pod \"payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42\" (UID: \"baba4bae-13e2-43e7-a5af-ad81a475d96b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:03:09.277903 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.277834 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baba4bae-13e2-43e7-a5af-ad81a475d96b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42\" (UID: \"baba4bae-13e2-43e7-a5af-ad81a475d96b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:03:09.287193 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.287172 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8654\" (UniqueName: \"kubernetes.io/projected/baba4bae-13e2-43e7-a5af-ad81a475d96b-kube-api-access-g8654\") pod \"payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42\" (UID: \"baba4bae-13e2-43e7-a5af-ad81a475d96b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:03:09.378078 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.377995 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ace602e-4cae-4e21-8b8c-efff41247ee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv\" (UID: \"1ace602e-4cae-4e21-8b8c-efff41247ee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:03:09.378078 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.378035 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkk5\" (UniqueName: \"kubernetes.io/projected/1ace602e-4cae-4e21-8b8c-efff41247ee7-kube-api-access-dwkk5\") pod \"payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv\" (UID: \"1ace602e-4cae-4e21-8b8c-efff41247ee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:03:09.378775 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.378753 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ace602e-4cae-4e21-8b8c-efff41247ee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv\" (UID: \"1ace602e-4cae-4e21-8b8c-efff41247ee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:03:09.387125 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.387101 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkk5\" (UniqueName: \"kubernetes.io/projected/1ace602e-4cae-4e21-8b8c-efff41247ee7-kube-api-access-dwkk5\") pod \"payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv\" (UID: \"1ace602e-4cae-4e21-8b8c-efff41247ee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:03:09.458311 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.458281 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:03:09.528350 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.528319 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:03:09.719911 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.719875 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" event={"ID":"cf406175-8413-4320-9043-0188ffc52141","Type":"ContainerStarted","Data":"9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb"} Sep 14 10:03:09.720167 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.719941 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" event={"ID":"cf406175-8413-4320-9043-0188ffc52141","Type":"ContainerStarted","Data":"e8007896ba77f22e48a8cf81eca8911151168d597f662cdaf57b9a02c55d1294"} Sep 14 10:03:09.720167 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.719975 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:09.730055 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.730030 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42"] Sep 14 10:03:09.731302 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:03:09.731281 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaba4bae_13e2_43e7_a5af_ad81a475d96b.slice/crio-18135673a745f4ce5ca1f12fa51579efa123bcaa5a1b0cb4ca185d94aa330d9e WatchSource:0}: Error finding container 18135673a745f4ce5ca1f12fa51579efa123bcaa5a1b0cb4ca185d94aa330d9e: Status 404 returned error can't find the container with id 18135673a745f4ce5ca1f12fa51579efa123bcaa5a1b0cb4ca185d94aa330d9e Sep 14 10:03:09.742005 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.741958 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" podStartSLOduration=1.313746531 podStartE2EDuration="1.741939294s" totalImagesPullingTime="428.192763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:03:09.739050771 +0000 UTC m=+1206.434176077" watchObservedRunningTime="2026-09-14 10:03:09.741939294 +0000 UTC m=+1206.437064600" Sep 14 10:03:09.811570 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:09.811545 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv"] Sep 14 10:03:09.813710 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:03:09.813682 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ace602e_4cae_4e21_8b8c_efff41247ee7.slice/crio-18b21a2bd591e265953c19061ef7df0640f1a95c29c6479f240c57229c8c6399 WatchSource:0}: Error finding container 18b21a2bd591e265953c19061ef7df0640f1a95c29c6479f240c57229c8c6399: Status 404 returned error can't find the container with id 18b21a2bd591e265953c19061ef7df0640f1a95c29c6479f240c57229c8c6399 Sep 14 10:03:10.727962 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:10.727923 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" event={"ID":"1ace602e-4cae-4e21-8b8c-efff41247ee7","Type":"ContainerStarted","Data":"138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c"} Sep 14 10:03:10.728420 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:10.727986 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" event={"ID":"1ace602e-4cae-4e21-8b8c-efff41247ee7","Type":"ContainerStarted","Data":"18b21a2bd591e265953c19061ef7df0640f1a95c29c6479f240c57229c8c6399"} Sep 14 10:03:10.728697 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:10.728681 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:03:10.731305 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:10.731241 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" event={"ID":"baba4bae-13e2-43e7-a5af-ad81a475d96b","Type":"ContainerStarted","Data":"c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b"} Sep 14 10:03:10.731305 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:10.731292 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" event={"ID":"baba4bae-13e2-43e7-a5af-ad81a475d96b","Type":"ContainerStarted","Data":"18135673a745f4ce5ca1f12fa51579efa123bcaa5a1b0cb4ca185d94aa330d9e"} Sep 14 10:03:10.731498 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:10.731321 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:03:10.746218 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:10.745750 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" podStartSLOduration=1.745733982 podStartE2EDuration="1.745733982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:03:10.745346659 +0000 UTC m=+1207.440471963" watchObservedRunningTime="2026-09-14 10:03:10.745733982 +0000 UTC m=+1207.440859288" Sep 14 10:03:10.763592 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:10.763546 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" podStartSLOduration=1.763532039 podStartE2EDuration="1.763532039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:03:10.761552811 +0000 UTC m=+1207.456678120" watchObservedRunningTime="2026-09-14 10:03:10.763532039 +0000 UTC m=+1207.458657348" Sep 14 10:03:15.737194 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:15.737162 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:03:21.740328 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:21.740298 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:03:22.745372 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:22.745343 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:03:44.482829 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.482795 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf"] Sep 14 10:03:44.487919 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.487897 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2"] Sep 14 10:03:44.488073 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.488031 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:03:44.490822 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.490797 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-9e19b7\"" Sep 14 10:03:44.491381 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.491365 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:03:44.496794 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.496769 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf"] Sep 14 10:03:44.506510 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.506489 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2"] Sep 14 10:03:44.519094 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.519062 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwgdg\" (UniqueName: \"kubernetes.io/projected/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-kube-api-access-gwgdg\") pod \"payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2\" (UID: \"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:03:44.519275 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.519112 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2\" (UID: \"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:03:44.519275 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.519241 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf\" (UID: \"3badb610-7f4b-4f50-a8d7-8c54f955ee1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:03:44.519398 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.519271 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8kvq\" (UniqueName: \"kubernetes.io/projected/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-kube-api-access-n8kvq\") pod \"payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf\" (UID: \"3badb610-7f4b-4f50-a8d7-8c54f955ee1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:03:44.620877 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.620835 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf\" (UID: \"3badb610-7f4b-4f50-a8d7-8c54f955ee1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:03:44.620877 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.620873 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8kvq\" (UniqueName: \"kubernetes.io/projected/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-kube-api-access-n8kvq\") pod \"payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf\" (UID: \"3badb610-7f4b-4f50-a8d7-8c54f955ee1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:03:44.621135 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.620938 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwgdg\" (UniqueName: \"kubernetes.io/projected/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-kube-api-access-gwgdg\") pod \"payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2\" (UID: \"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:03:44.621135 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.620960 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2\" (UID: \"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:03:44.621690 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.621666 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2\" (UID: \"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:03:44.621804 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.621668 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf\" (UID: \"3badb610-7f4b-4f50-a8d7-8c54f955ee1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:03:44.633578 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.633550 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwgdg\" (UniqueName: \"kubernetes.io/projected/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-kube-api-access-gwgdg\") pod \"payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2\" (UID: \"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:03:44.633723 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.633687 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8kvq\" (UniqueName: \"kubernetes.io/projected/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-kube-api-access-n8kvq\") pod \"payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf\" (UID: \"3badb610-7f4b-4f50-a8d7-8c54f955ee1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:03:44.802249 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.802169 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:03:44.810244 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:44.810226 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:03:45.093051 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:03:45.093011 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f2b5fb_87d9_4472_9a7a_3ccaec51e3e4.slice/crio-787caaadd0ee884f0c5a978b6e07b66993c6dbfc37648f23925ba7a8eff54f90 WatchSource:0}: Error finding container 787caaadd0ee884f0c5a978b6e07b66993c6dbfc37648f23925ba7a8eff54f90: Status 404 returned error can't find the container with id 787caaadd0ee884f0c5a978b6e07b66993c6dbfc37648f23925ba7a8eff54f90 Sep 14 10:03:45.093464 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:03:45.093441 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3badb610_7f4b_4f50_a8d7_8c54f955ee1b.slice/crio-99260b9cf5ab97d6ec192b8b0446fab0534a6eb7f96cf77f6a892d74e13f2aba WatchSource:0}: Error finding container 99260b9cf5ab97d6ec192b8b0446fab0534a6eb7f96cf77f6a892d74e13f2aba: Status 404 returned error can't find the container with id 99260b9cf5ab97d6ec192b8b0446fab0534a6eb7f96cf77f6a892d74e13f2aba Sep 14 10:03:45.093566 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:45.093449 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf"] Sep 14 10:03:45.096863 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:45.096844 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2"] Sep 14 10:03:45.930415 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:45.930379 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" event={"ID":"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4","Type":"ContainerStarted","Data":"52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9"} Sep 14 10:03:45.930950 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:45.930442 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" event={"ID":"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4","Type":"ContainerStarted","Data":"787caaadd0ee884f0c5a978b6e07b66993c6dbfc37648f23925ba7a8eff54f90"} Sep 14 10:03:45.930950 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:45.930477 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:03:45.932159 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:45.932128 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" event={"ID":"3badb610-7f4b-4f50-a8d7-8c54f955ee1b","Type":"ContainerStarted","Data":"09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec"} Sep 14 10:03:45.932307 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:45.932188 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:03:45.932307 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:45.932213 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" event={"ID":"3badb610-7f4b-4f50-a8d7-8c54f955ee1b","Type":"ContainerStarted","Data":"99260b9cf5ab97d6ec192b8b0446fab0534a6eb7f96cf77f6a892d74e13f2aba"} Sep 14 10:03:46.006574 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:46.006524 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" podStartSLOduration=2.006507502 podStartE2EDuration="2.006507502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:03:46.006334693 +0000 UTC m=+1242.701459997" watchObservedRunningTime="2026-09-14 10:03:46.006507502 +0000 UTC m=+1242.701632806" Sep 14 10:03:46.007967 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:46.007924 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" podStartSLOduration=2.007913328 podStartE2EDuration="2.007913328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:03:45.968445259 +0000 UTC m=+1242.663570563" watchObservedRunningTime="2026-09-14 10:03:46.007913328 +0000 UTC m=+1242.703038698" Sep 14 10:03:56.941186 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:56.941096 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:03:56.941534 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:03:56.941259 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:04:16.962804 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:16.962738 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x"] Sep 14 10:04:16.963410 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:16.962978 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" podUID="4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe" containerName="maas-api" containerID="cri-o://b736b40fdd723e0400e2e29164f0631d51b980fa3bdd8cbb0b780b36689c5b57" gracePeriod=30 Sep 14 10:04:17.038727 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.038693 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp"] Sep 14 10:04:17.039004 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.038977 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" podUID="a9b07a80-34ab-4958-a61b-a94451edbb1b" containerName="payload-processing" containerID="cri-o://858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40" gracePeriod=30 Sep 14 10:04:17.046534 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.046507 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6"] Sep 14 10:04:17.046885 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.046769 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" podUID="b3e7cc8a-91aa-4dc7-a649-c329927f4f62" containerName="payload-pre-processing" containerID="cri-o://7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772" gracePeriod=30 Sep 14 10:04:17.118198 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.118168 2550 generic.go:356] "Generic (PLEG): container finished" podID="4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe" containerID="b736b40fdd723e0400e2e29164f0631d51b980fa3bdd8cbb0b780b36689c5b57" exitCode=0 Sep 14 10:04:17.118344 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.118238 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" event={"ID":"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe","Type":"ContainerDied","Data":"b736b40fdd723e0400e2e29164f0631d51b980fa3bdd8cbb0b780b36689c5b57"} Sep 14 10:04:17.364429 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.364408 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 10:04:17.368371 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.368343 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 10:04:17.371875 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.371855 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 10:04:17.535292 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.535183 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgrsh\" (UniqueName: \"kubernetes.io/projected/a9b07a80-34ab-4958-a61b-a94451edbb1b-kube-api-access-xgrsh\") pod \"a9b07a80-34ab-4958-a61b-a94451edbb1b\" (UID: \"a9b07a80-34ab-4958-a61b-a94451edbb1b\") " Sep 14 10:04:17.535949 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.535922 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b07a80-34ab-4958-a61b-a94451edbb1b-plugins-config-volume\") pod \"a9b07a80-34ab-4958-a61b-a94451edbb1b\" (UID: \"a9b07a80-34ab-4958-a61b-a94451edbb1b\") " Sep 14 10:04:17.536101 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.535961 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq98n\" (UniqueName: \"kubernetes.io/projected/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-kube-api-access-xq98n\") pod \"b3e7cc8a-91aa-4dc7-a649-c329927f4f62\" (UID: \"b3e7cc8a-91aa-4dc7-a649-c329927f4f62\") " Sep 14 10:04:17.536101 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.536030 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d96dz\" (UniqueName: \"kubernetes.io/projected/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-kube-api-access-d96dz\") pod \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " Sep 14 10:04:17.536236 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.536102 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-maas-api-tls\") pod \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " Sep 14 10:04:17.536291 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.536233 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-metrics-certs\") pod \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " Sep 14 10:04:17.536291 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.536266 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-plugins-config-volume\") pod \"b3e7cc8a-91aa-4dc7-a649-c329927f4f62\" (UID: \"b3e7cc8a-91aa-4dc7-a649-c329927f4f62\") " Sep 14 10:04:17.536402 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.536297 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-openshift-service-ca\") pod \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\" (UID: \"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe\") " Sep 14 10:04:17.536402 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.536374 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b07a80-34ab-4958-a61b-a94451edbb1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9b07a80-34ab-4958-a61b-a94451edbb1b" (UID: "a9b07a80-34ab-4958-a61b-a94451edbb1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:04:17.536711 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.536691 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b07a80-34ab-4958-a61b-a94451edbb1b-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:17.536853 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.536772 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3e7cc8a-91aa-4dc7-a649-c329927f4f62" (UID: "b3e7cc8a-91aa-4dc7-a649-c329927f4f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:04:17.536853 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.536817 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe" (UID: "4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:04:17.538061 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.538041 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b07a80-34ab-4958-a61b-a94451edbb1b-kube-api-access-xgrsh" (OuterVolumeSpecName: "kube-api-access-xgrsh") pod "a9b07a80-34ab-4958-a61b-a94451edbb1b" (UID: "a9b07a80-34ab-4958-a61b-a94451edbb1b"). InnerVolumeSpecName "kube-api-access-xgrsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:04:17.538395 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.538373 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe" (UID: "4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:04:17.538709 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.538690 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-kube-api-access-d96dz" (OuterVolumeSpecName: "kube-api-access-d96dz") pod "4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe" (UID: "4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe"). InnerVolumeSpecName "kube-api-access-d96dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:04:17.538795 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.538754 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-kube-api-access-xq98n" (OuterVolumeSpecName: "kube-api-access-xq98n") pod "b3e7cc8a-91aa-4dc7-a649-c329927f4f62" (UID: "b3e7cc8a-91aa-4dc7-a649-c329927f4f62"). InnerVolumeSpecName "kube-api-access-xq98n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:04:17.538853 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.538833 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe" (UID: "4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:04:17.637924 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.637885 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgrsh\" (UniqueName: \"kubernetes.io/projected/a9b07a80-34ab-4958-a61b-a94451edbb1b-kube-api-access-xgrsh\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:17.637924 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.637917 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq98n\" (UniqueName: \"kubernetes.io/projected/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-kube-api-access-xq98n\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:17.637924 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.637931 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d96dz\" (UniqueName: \"kubernetes.io/projected/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-kube-api-access-d96dz\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:17.638169 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.637944 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:17.638169 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.637955 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:17.638169 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.637965 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e7cc8a-91aa-4dc7-a649-c329927f4f62-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:17.638169 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:17.637978 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:18.125837 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.125752 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" Sep 14 10:04:18.125837 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.125770 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x" event={"ID":"4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe","Type":"ContainerDied","Data":"cb05eb796f881e1cc7ed046f53326e5f068b8698feabe90daeede12ec778ae04"} Sep 14 10:04:18.125837 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.125832 2550 scope.go:117] "RemoveContainer" containerID="b736b40fdd723e0400e2e29164f0631d51b980fa3bdd8cbb0b780b36689c5b57" Sep 14 10:04:18.127482 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.127456 2550 generic.go:356] "Generic (PLEG): container finished" podID="a9b07a80-34ab-4958-a61b-a94451edbb1b" containerID="858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40" exitCode=0 Sep 14 10:04:18.127590 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.127509 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" Sep 14 10:04:18.127708 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.127488 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" event={"ID":"a9b07a80-34ab-4958-a61b-a94451edbb1b","Type":"ContainerDied","Data":"858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40"} Sep 14 10:04:18.127776 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.127697 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp" event={"ID":"a9b07a80-34ab-4958-a61b-a94451edbb1b","Type":"ContainerDied","Data":"fa758f208a96dbf0f55560d4d6ebfbc34188c0b065c01b8eee57bf06f2b2dbbc"} Sep 14 10:04:18.129262 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.129236 2550 generic.go:356] "Generic (PLEG): container finished" podID="b3e7cc8a-91aa-4dc7-a649-c329927f4f62" containerID="7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772" exitCode=0 Sep 14 10:04:18.129367 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.129279 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" Sep 14 10:04:18.129367 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.129290 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" event={"ID":"b3e7cc8a-91aa-4dc7-a649-c329927f4f62","Type":"ContainerDied","Data":"7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772"} Sep 14 10:04:18.129367 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.129327 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6" event={"ID":"b3e7cc8a-91aa-4dc7-a649-c329927f4f62","Type":"ContainerDied","Data":"4f82f7db3c6c9a9be3799f30c72f91cd45567de799faa59abf1b116d037a0f14"} Sep 14 10:04:18.136928 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.136911 2550 scope.go:117] "RemoveContainer" containerID="858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40" Sep 14 10:04:18.145129 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.145115 2550 scope.go:117] "RemoveContainer" containerID="858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40" Sep 14 10:04:18.145384 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:04:18.145367 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40\": container with ID starting with 858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40 not found: ID does not exist" containerID="858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40" Sep 14 10:04:18.145455 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.145396 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40"} err="failed to get container status \"858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40\": rpc error: code = NotFound desc = could not find container \"858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40\": container with ID starting with 858161d5a01c86135e93715fc57bd38dbf2421121e24003e354178b5b81bbd40 not found: ID does not exist" Sep 14 10:04:18.145455 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.145421 2550 scope.go:117] "RemoveContainer" containerID="7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772" Sep 14 10:04:18.154232 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.154213 2550 scope.go:117] "RemoveContainer" containerID="7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772" Sep 14 10:04:18.154487 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:04:18.154464 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772\": container with ID starting with 7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772 not found: ID does not exist" containerID="7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772" Sep 14 10:04:18.154529 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.154497 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772"} err="failed to get container status \"7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772\": rpc error: code = NotFound desc = could not find container \"7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772\": container with ID starting with 7fc4b5b94d137a926eaa3d87da05dd28f240f87dc866db09409ecf0d3592c772 not found: ID does not exist" Sep 14 10:04:18.154990 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.154974 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6"] Sep 14 10:04:18.161965 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.161938 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e60f3-5c5dfbf758-w6cp6"] Sep 14 10:04:18.171211 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.171190 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp"] Sep 14 10:04:18.175092 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.175074 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e60f3-6bbdf465d9-pgrvp"] Sep 14 10:04:18.192137 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.185738 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x"] Sep 14 10:04:18.193304 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:18.193282 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e60f3-6b44bd7545-lkf8x"] Sep 14 10:04:19.826042 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:19.826004 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe" path="/var/lib/kubelet/pods/4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe/volumes" Sep 14 10:04:19.826417 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:19.826407 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b07a80-34ab-4958-a61b-a94451edbb1b" path="/var/lib/kubelet/pods/a9b07a80-34ab-4958-a61b-a94451edbb1b/volumes" Sep 14 10:04:19.826854 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:19.826834 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3e7cc8a-91aa-4dc7-a649-c329927f4f62" path="/var/lib/kubelet/pods/b3e7cc8a-91aa-4dc7-a649-c329927f4f62/volumes" Sep 14 10:04:31.168007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.165797 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m"] Sep 14 10:04:31.168007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.166423 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3e7cc8a-91aa-4dc7-a649-c329927f4f62" containerName="payload-pre-processing" Sep 14 10:04:31.168007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.166443 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3e7cc8a-91aa-4dc7-a649-c329927f4f62" containerName="payload-pre-processing" Sep 14 10:04:31.168007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.166473 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b07a80-34ab-4958-a61b-a94451edbb1b" containerName="payload-processing" Sep 14 10:04:31.168007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.166482 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b07a80-34ab-4958-a61b-a94451edbb1b" containerName="payload-processing" Sep 14 10:04:31.168007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.166506 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe" containerName="maas-api" Sep 14 10:04:31.168007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.166513 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe" containerName="maas-api" Sep 14 10:04:31.168007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.166592 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b07a80-34ab-4958-a61b-a94451edbb1b" containerName="payload-processing" Sep 14 10:04:31.168007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.166610 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3e7cc8a-91aa-4dc7-a649-c329927f4f62" containerName="payload-pre-processing" Sep 14 10:04:31.168007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.166620 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa3e4f7-68b3-46ae-b8b0-4f7fe79b96fe" containerName="maas-api" Sep 14 10:04:31.171715 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.171692 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.174905 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.174878 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-75017ebc-serving-cert\"" Sep 14 10:04:31.183935 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.183829 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m"] Sep 14 10:04:31.264802 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.264773 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-metrics-certs\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.264961 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.264824 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5wms\" (UniqueName: \"kubernetes.io/projected/5c244ab8-5b4b-4100-b40f-032440e6b89d-kube-api-access-h5wms\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.264961 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.264860 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c244ab8-5b4b-4100-b40f-032440e6b89d-openshift-service-ca\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.264961 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.264902 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-maas-api-tls\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.366014 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.365979 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-maas-api-tls\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.366154 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.366110 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-metrics-certs\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.366154 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:04:31.366115 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-serving-cert: secret "maas-api-e2e-mt-75017ebc-serving-cert" not found Sep 14 10:04:31.366154 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.366144 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5wms\" (UniqueName: \"kubernetes.io/projected/5c244ab8-5b4b-4100-b40f-032440e6b89d-kube-api-access-h5wms\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.366321 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:04:31.366186 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-maas-api-tls podName:5c244ab8-5b4b-4100-b40f-032440e6b89d nodeName:}" failed. No retries permitted until 2026-09-14 10:04:31.86616755 +0000 UTC m=+1288.561292845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-maas-api-tls") pod "maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" (UID: "5c244ab8-5b4b-4100-b40f-032440e6b89d") : secret "maas-api-e2e-mt-75017ebc-serving-cert" not found Sep 14 10:04:31.366321 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.366205 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c244ab8-5b4b-4100-b40f-032440e6b89d-openshift-service-ca\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.366893 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.366875 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c244ab8-5b4b-4100-b40f-032440e6b89d-openshift-service-ca\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.368583 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.368561 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-metrics-certs\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.377238 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.377214 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5wms\" (UniqueName: \"kubernetes.io/projected/5c244ab8-5b4b-4100-b40f-032440e6b89d-kube-api-access-h5wms\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.870720 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.870678 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-maas-api-tls\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:31.873085 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:31.873059 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-maas-api-tls\") pod \"maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:32.084797 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:32.084760 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:32.363732 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:32.363703 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m"] Sep 14 10:04:32.365503 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:04:32.365476 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c244ab8_5b4b_4100_b40f_032440e6b89d.slice/crio-33a80eac5b9310c5640e58af03deb44f812c04d6c358d24549ea767a168828bd WatchSource:0}: Error finding container 33a80eac5b9310c5640e58af03deb44f812c04d6c358d24549ea767a168828bd: Status 404 returned error can't find the container with id 33a80eac5b9310c5640e58af03deb44f812c04d6c358d24549ea767a168828bd Sep 14 10:04:33.219887 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:33.219852 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" event={"ID":"5c244ab8-5b4b-4100-b40f-032440e6b89d","Type":"ContainerStarted","Data":"3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4"} Sep 14 10:04:33.220098 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:33.219907 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" event={"ID":"5c244ab8-5b4b-4100-b40f-032440e6b89d","Type":"ContainerStarted","Data":"33a80eac5b9310c5640e58af03deb44f812c04d6c358d24549ea767a168828bd"} Sep 14 10:04:33.220098 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:33.220000 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:33.293618 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:33.293560 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" podStartSLOduration=1.9043143900000001 podStartE2EDuration="2.293544978s" totalImagesPullingTime="389.230588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:33.291852073 +0000 UTC m=+1289.986977378" watchObservedRunningTime="2026-09-14 10:04:33.293544978 +0000 UTC m=+1289.988670282" Sep 14 10:04:35.540688 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.540651 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2"] Sep 14 10:04:35.541069 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.540839 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" podUID="afb2dcff-114f-48d9-9baf-401b6ac59183" containerName="maas-api" containerID="cri-o://7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c" gracePeriod=30 Sep 14 10:04:35.631592 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.629459 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk"] Sep 14 10:04:35.634228 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.634202 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24873d-29823000-4j9tk"] Sep 14 10:04:35.658241 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.658209 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf"] Sep 14 10:04:35.658539 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.658491 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" podUID="0283d4d0-2868-4754-9323-53af7d7ded81" containerName="payload-processing" containerID="cri-o://c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560" gracePeriod=30 Sep 14 10:04:35.810332 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.810301 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 10:04:35.827104 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.826845 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95db1855-1a94-47a1-a5a6-856b8a49ff5c" path="/var/lib/kubelet/pods/95db1855-1a94-47a1-a5a6-856b8a49ff5c/volumes" Sep 14 10:04:35.909787 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.909756 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk2gs\" (UniqueName: \"kubernetes.io/projected/afb2dcff-114f-48d9-9baf-401b6ac59183-kube-api-access-fk2gs\") pod \"afb2dcff-114f-48d9-9baf-401b6ac59183\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " Sep 14 10:04:35.909956 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.909797 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb2dcff-114f-48d9-9baf-401b6ac59183-openshift-service-ca\") pod \"afb2dcff-114f-48d9-9baf-401b6ac59183\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " Sep 14 10:04:35.909956 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.909848 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-maas-api-tls\") pod \"afb2dcff-114f-48d9-9baf-401b6ac59183\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " Sep 14 10:04:35.910042 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.910013 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-metrics-certs\") pod \"afb2dcff-114f-48d9-9baf-401b6ac59183\" (UID: \"afb2dcff-114f-48d9-9baf-401b6ac59183\") " Sep 14 10:04:35.910253 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.910230 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb2dcff-114f-48d9-9baf-401b6ac59183-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "afb2dcff-114f-48d9-9baf-401b6ac59183" (UID: "afb2dcff-114f-48d9-9baf-401b6ac59183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:04:35.910442 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.910420 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb2dcff-114f-48d9-9baf-401b6ac59183-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:35.912048 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.912020 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "afb2dcff-114f-48d9-9baf-401b6ac59183" (UID: "afb2dcff-114f-48d9-9baf-401b6ac59183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:04:35.912162 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.912056 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "afb2dcff-114f-48d9-9baf-401b6ac59183" (UID: "afb2dcff-114f-48d9-9baf-401b6ac59183"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:04:35.912162 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.912107 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb2dcff-114f-48d9-9baf-401b6ac59183-kube-api-access-fk2gs" (OuterVolumeSpecName: "kube-api-access-fk2gs") pod "afb2dcff-114f-48d9-9baf-401b6ac59183" (UID: "afb2dcff-114f-48d9-9baf-401b6ac59183"). InnerVolumeSpecName "kube-api-access-fk2gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:04:35.932561 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:35.932542 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 10:04:36.011343 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.011311 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zchbl\" (UniqueName: \"kubernetes.io/projected/0283d4d0-2868-4754-9323-53af7d7ded81-kube-api-access-zchbl\") pod \"0283d4d0-2868-4754-9323-53af7d7ded81\" (UID: \"0283d4d0-2868-4754-9323-53af7d7ded81\") " Sep 14 10:04:36.011537 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.011377 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0283d4d0-2868-4754-9323-53af7d7ded81-plugins-config-volume\") pod \"0283d4d0-2868-4754-9323-53af7d7ded81\" (UID: \"0283d4d0-2868-4754-9323-53af7d7ded81\") " Sep 14 10:04:36.011604 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.011593 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:36.011702 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.011604 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk2gs\" (UniqueName: \"kubernetes.io/projected/afb2dcff-114f-48d9-9baf-401b6ac59183-kube-api-access-fk2gs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:36.011702 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.011648 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afb2dcff-114f-48d9-9baf-401b6ac59183-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:36.011809 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.011749 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0283d4d0-2868-4754-9323-53af7d7ded81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0283d4d0-2868-4754-9323-53af7d7ded81" (UID: "0283d4d0-2868-4754-9323-53af7d7ded81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:04:36.013361 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.013338 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0283d4d0-2868-4754-9323-53af7d7ded81-kube-api-access-zchbl" (OuterVolumeSpecName: "kube-api-access-zchbl") pod "0283d4d0-2868-4754-9323-53af7d7ded81" (UID: "0283d4d0-2868-4754-9323-53af7d7ded81"). InnerVolumeSpecName "kube-api-access-zchbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:04:36.113131 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.113055 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zchbl\" (UniqueName: \"kubernetes.io/projected/0283d4d0-2868-4754-9323-53af7d7ded81-kube-api-access-zchbl\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:36.113131 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.113083 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0283d4d0-2868-4754-9323-53af7d7ded81-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:04:36.237469 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.237431 2550 generic.go:356] "Generic (PLEG): container finished" podID="0283d4d0-2868-4754-9323-53af7d7ded81" containerID="c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560" exitCode=0 Sep 14 10:04:36.237626 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.237481 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" Sep 14 10:04:36.237626 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.237512 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" event={"ID":"0283d4d0-2868-4754-9323-53af7d7ded81","Type":"ContainerDied","Data":"c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560"} Sep 14 10:04:36.237626 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.237568 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf" event={"ID":"0283d4d0-2868-4754-9323-53af7d7ded81","Type":"ContainerDied","Data":"fbe359074a9ca4405f5e7bbb9eced082ef51fe7c6125855ccbd1075823b712e7"} Sep 14 10:04:36.237626 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.237594 2550 scope.go:117] "RemoveContainer" containerID="c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560" Sep 14 10:04:36.239218 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.239195 2550 generic.go:356] "Generic (PLEG): container finished" podID="afb2dcff-114f-48d9-9baf-401b6ac59183" containerID="7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c" exitCode=0 Sep 14 10:04:36.239311 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.239236 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" Sep 14 10:04:36.239311 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.239248 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" event={"ID":"afb2dcff-114f-48d9-9baf-401b6ac59183","Type":"ContainerDied","Data":"7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c"} Sep 14 10:04:36.239311 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.239286 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2" event={"ID":"afb2dcff-114f-48d9-9baf-401b6ac59183","Type":"ContainerDied","Data":"33ed5151c66688ff04a9922d759643707c3ecd32fd78979ca9e673ff4ceef1cf"} Sep 14 10:04:36.247561 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.247521 2550 scope.go:117] "RemoveContainer" containerID="c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560" Sep 14 10:04:36.247861 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:04:36.247842 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560\": container with ID starting with c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560 not found: ID does not exist" containerID="c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560" Sep 14 10:04:36.247965 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.247872 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560"} err="failed to get container status \"c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560\": rpc error: code = NotFound desc = could not find container \"c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560\": container with ID starting with c3808f9d33845055134c274358f3884c48f4b3448fa6502dca14045d4285a560 not found: ID does not exist" Sep 14 10:04:36.247965 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.247894 2550 scope.go:117] "RemoveContainer" containerID="7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c" Sep 14 10:04:36.256325 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.256311 2550 scope.go:117] "RemoveContainer" containerID="7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c" Sep 14 10:04:36.256601 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:04:36.256583 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c\": container with ID starting with 7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c not found: ID does not exist" containerID="7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c" Sep 14 10:04:36.256780 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.256608 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c"} err="failed to get container status \"7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c\": rpc error: code = NotFound desc = could not find container \"7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c\": container with ID starting with 7fe87ffdee8d34c56a0386f8aa4033c4527348b6bf3f0cebb6564c735b78106c not found: ID does not exist" Sep 14 10:04:36.266206 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.266188 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf"] Sep 14 10:04:36.272782 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.272763 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24873d-6f779f8d75-bzdrf"] Sep 14 10:04:36.290845 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.290818 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2"] Sep 14 10:04:36.293467 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:36.293446 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24873d-5cbff5dd78-xvqf2"] Sep 14 10:04:37.826465 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:37.826432 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0283d4d0-2868-4754-9323-53af7d7ded81" path="/var/lib/kubelet/pods/0283d4d0-2868-4754-9323-53af7d7ded81/volumes" Sep 14 10:04:37.826974 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:37.826883 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afb2dcff-114f-48d9-9baf-401b6ac59183" path="/var/lib/kubelet/pods/afb2dcff-114f-48d9-9baf-401b6ac59183/volumes" Sep 14 10:04:39.230165 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:39.230137 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:04:53.976695 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:53.976662 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj"] Sep 14 10:04:53.977308 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:53.977149 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb2dcff-114f-48d9-9baf-401b6ac59183" containerName="maas-api" Sep 14 10:04:53.977308 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:53.977163 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb2dcff-114f-48d9-9baf-401b6ac59183" containerName="maas-api" Sep 14 10:04:53.977308 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:53.977179 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0283d4d0-2868-4754-9323-53af7d7ded81" containerName="payload-processing" Sep 14 10:04:53.977308 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:53.977187 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0283d4d0-2868-4754-9323-53af7d7ded81" containerName="payload-processing" Sep 14 10:04:53.977308 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:53.977253 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb2dcff-114f-48d9-9baf-401b6ac59183" containerName="maas-api" Sep 14 10:04:53.977308 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:53.977261 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0283d4d0-2868-4754-9323-53af7d7ded81" containerName="payload-processing" Sep 14 10:04:53.981777 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:53.981754 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:53.984190 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:53.984166 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-51aefd-serving-cert\"" Sep 14 10:04:53.994477 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:53.994089 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj"] Sep 14 10:04:54.091469 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.091436 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.091650 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.091492 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlltl\" (UniqueName: \"kubernetes.io/projected/563cac49-ae20-4251-8a35-1e7f0aea1846-kube-api-access-dlltl\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.091650 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.091593 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563cac49-ae20-4251-8a35-1e7f0aea1846-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.091748 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.091676 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.192810 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.192777 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563cac49-ae20-4251-8a35-1e7f0aea1846-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.192989 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.192827 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.192989 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.192958 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.193106 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.193008 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlltl\" (UniqueName: \"kubernetes.io/projected/563cac49-ae20-4251-8a35-1e7f0aea1846-kube-api-access-dlltl\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.193106 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:04:54.193081 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-serving-cert: secret "maas-api-e2e-shared-a-w3-51aefd-serving-cert" not found Sep 14 10:04:54.193214 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:04:54.193154 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-maas-api-tls podName:563cac49-ae20-4251-8a35-1e7f0aea1846 nodeName:}" failed. No retries permitted until 2026-09-14 10:04:54.693133849 +0000 UTC m=+1311.388259155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-maas-api-tls") pod "maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" (UID: "563cac49-ae20-4251-8a35-1e7f0aea1846") : secret "maas-api-e2e-shared-a-w3-51aefd-serving-cert" not found Sep 14 10:04:54.193517 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.193493 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563cac49-ae20-4251-8a35-1e7f0aea1846-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.195376 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.195358 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.203652 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.203608 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlltl\" (UniqueName: \"kubernetes.io/projected/563cac49-ae20-4251-8a35-1e7f0aea1846-kube-api-access-dlltl\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.698749 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.698707 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.701348 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.701323 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.856366 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.856329 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9"] Sep 14 10:04:54.860245 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.860219 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:04:54.863214 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.863196 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-51aefd\"" Sep 14 10:04:54.874491 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.874471 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9"] Sep 14 10:04:54.895776 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.895752 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:54.910078 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.910057 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf"] Sep 14 10:04:54.913964 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.913947 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:04:54.917057 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.917038 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-51aefd-dockercfg-rkrld\"" Sep 14 10:04:54.931799 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:54.931626 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf"] Sep 14 10:04:55.001991 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.001955 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqzkd\" (UniqueName: \"kubernetes.io/projected/6c3d860d-6ef8-45f8-9cae-789f39846cdf-kube-api-access-pqzkd\") pod \"payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9\" (UID: \"6c3d860d-6ef8-45f8-9cae-789f39846cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:04:55.002362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.002003 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3d860d-6ef8-45f8-9cae-789f39846cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9\" (UID: \"6c3d860d-6ef8-45f8-9cae-789f39846cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:04:55.103054 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.103018 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zbmw\" (UniqueName: \"kubernetes.io/projected/19989a8b-8523-4b16-90a2-db5a9b372f2e-kube-api-access-6zbmw\") pod \"payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf\" (UID: \"19989a8b-8523-4b16-90a2-db5a9b372f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:04:55.103216 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.103088 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqzkd\" (UniqueName: \"kubernetes.io/projected/6c3d860d-6ef8-45f8-9cae-789f39846cdf-kube-api-access-pqzkd\") pod \"payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9\" (UID: \"6c3d860d-6ef8-45f8-9cae-789f39846cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:04:55.103216 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.103135 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19989a8b-8523-4b16-90a2-db5a9b372f2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf\" (UID: \"19989a8b-8523-4b16-90a2-db5a9b372f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:04:55.103216 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.103172 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3d860d-6ef8-45f8-9cae-789f39846cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9\" (UID: \"6c3d860d-6ef8-45f8-9cae-789f39846cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:04:55.103926 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.103905 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3d860d-6ef8-45f8-9cae-789f39846cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9\" (UID: \"6c3d860d-6ef8-45f8-9cae-789f39846cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:04:55.112188 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.112166 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqzkd\" (UniqueName: \"kubernetes.io/projected/6c3d860d-6ef8-45f8-9cae-789f39846cdf-kube-api-access-pqzkd\") pod \"payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9\" (UID: \"6c3d860d-6ef8-45f8-9cae-789f39846cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:04:55.171109 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.171081 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:04:55.186561 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.186536 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj"] Sep 14 10:04:55.187281 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:04:55.187257 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563cac49_ae20_4251_8a35_1e7f0aea1846.slice/crio-de39135b2289115b0e82fb830ad935425e41a487134f872c6a73befd7b0caa89 WatchSource:0}: Error finding container de39135b2289115b0e82fb830ad935425e41a487134f872c6a73befd7b0caa89: Status 404 returned error can't find the container with id de39135b2289115b0e82fb830ad935425e41a487134f872c6a73befd7b0caa89 Sep 14 10:04:55.203816 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.203758 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19989a8b-8523-4b16-90a2-db5a9b372f2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf\" (UID: \"19989a8b-8523-4b16-90a2-db5a9b372f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:04:55.203908 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.203893 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbmw\" (UniqueName: \"kubernetes.io/projected/19989a8b-8523-4b16-90a2-db5a9b372f2e-kube-api-access-6zbmw\") pod \"payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf\" (UID: \"19989a8b-8523-4b16-90a2-db5a9b372f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:04:55.204382 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.204362 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19989a8b-8523-4b16-90a2-db5a9b372f2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf\" (UID: \"19989a8b-8523-4b16-90a2-db5a9b372f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:04:55.214094 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.213300 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbmw\" (UniqueName: \"kubernetes.io/projected/19989a8b-8523-4b16-90a2-db5a9b372f2e-kube-api-access-6zbmw\") pod \"payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf\" (UID: \"19989a8b-8523-4b16-90a2-db5a9b372f2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:04:55.234936 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.234298 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:04:55.350563 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.350528 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" event={"ID":"563cac49-ae20-4251-8a35-1e7f0aea1846","Type":"ContainerStarted","Data":"de39135b2289115b0e82fb830ad935425e41a487134f872c6a73befd7b0caa89"} Sep 14 10:04:55.456078 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.456054 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9"] Sep 14 10:04:55.457647 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:04:55.457605 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3d860d_6ef8_45f8_9cae_789f39846cdf.slice/crio-62a8bb66409014fca807812497719ddd8acc61fd9539aed5438d5dffca304cb2 WatchSource:0}: Error finding container 62a8bb66409014fca807812497719ddd8acc61fd9539aed5438d5dffca304cb2: Status 404 returned error can't find the container with id 62a8bb66409014fca807812497719ddd8acc61fd9539aed5438d5dffca304cb2 Sep 14 10:04:55.524143 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:04:55.524117 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19989a8b_8523_4b16_90a2_db5a9b372f2e.slice/crio-3cf81d8dcdacc40fa186a9f2dc03b0974ff1c6cf5cdda3731a81d94f7c85192c WatchSource:0}: Error finding container 3cf81d8dcdacc40fa186a9f2dc03b0974ff1c6cf5cdda3731a81d94f7c85192c: Status 404 returned error can't find the container with id 3cf81d8dcdacc40fa186a9f2dc03b0974ff1c6cf5cdda3731a81d94f7c85192c Sep 14 10:04:55.526057 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:55.526033 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf"] Sep 14 10:04:56.358098 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.358058 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" event={"ID":"563cac49-ae20-4251-8a35-1e7f0aea1846","Type":"ContainerStarted","Data":"5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549"} Sep 14 10:04:56.358564 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.358137 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:04:56.359875 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.359850 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" event={"ID":"6c3d860d-6ef8-45f8-9cae-789f39846cdf","Type":"ContainerStarted","Data":"74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0"} Sep 14 10:04:56.360010 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.359896 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" event={"ID":"6c3d860d-6ef8-45f8-9cae-789f39846cdf","Type":"ContainerStarted","Data":"62a8bb66409014fca807812497719ddd8acc61fd9539aed5438d5dffca304cb2"} Sep 14 10:04:56.360010 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.359919 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:04:56.361463 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.361444 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" event={"ID":"19989a8b-8523-4b16-90a2-db5a9b372f2e","Type":"ContainerStarted","Data":"60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a"} Sep 14 10:04:56.361541 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.361486 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" event={"ID":"19989a8b-8523-4b16-90a2-db5a9b372f2e","Type":"ContainerStarted","Data":"3cf81d8dcdacc40fa186a9f2dc03b0974ff1c6cf5cdda3731a81d94f7c85192c"} Sep 14 10:04:56.361541 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.361515 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:04:56.391862 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.391805 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" podStartSLOduration=3.006041528 podStartE2EDuration="3.39178694s" totalImagesPullingTime="385.745412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:56.391347382 +0000 UTC m=+1313.086472687" watchObservedRunningTime="2026-09-14 10:04:56.39178694 +0000 UTC m=+1313.086912246" Sep 14 10:04:56.435993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.435945 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" podStartSLOduration=2.435932266 podStartE2EDuration="2.435932266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:56.435506848 +0000 UTC m=+1313.130632166" watchObservedRunningTime="2026-09-14 10:04:56.435932266 +0000 UTC m=+1313.131057571" Sep 14 10:04:56.457556 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:04:56.457510 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" podStartSLOduration=2.457492703 podStartE2EDuration="2.457492703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:56.454494666 +0000 UTC m=+1313.149619971" watchObservedRunningTime="2026-09-14 10:04:56.457492703 +0000 UTC m=+1313.152618012" Sep 14 10:05:02.372238 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:02.372209 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:05:07.368857 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:07.368820 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:05:07.369303 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:07.369006 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:05:29.187240 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.187206 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk"] Sep 14 10:05:29.192644 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.192618 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:05:29.195284 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.195264 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f4d054-dockercfg-mwxlb\"" Sep 14 10:05:29.195392 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.195272 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f4d054\"" Sep 14 10:05:29.201958 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.201936 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk"] Sep 14 10:05:29.333019 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.332982 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2cl\" (UniqueName: \"kubernetes.io/projected/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-kube-api-access-4f2cl\") pod \"payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk\" (UID: \"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:05:29.333210 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.333142 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk\" (UID: \"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:05:29.434369 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.434334 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk\" (UID: \"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:05:29.434538 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.434416 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2cl\" (UniqueName: \"kubernetes.io/projected/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-kube-api-access-4f2cl\") pod \"payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk\" (UID: \"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:05:29.435197 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.435164 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk\" (UID: \"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:05:29.442254 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.442195 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2cl\" (UniqueName: \"kubernetes.io/projected/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-kube-api-access-4f2cl\") pod \"payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk\" (UID: \"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:05:29.505078 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.505058 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:05:29.778302 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:29.778273 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk"] Sep 14 10:05:29.780085 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:05:29.780061 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69e38b7_f4e0_4ae0_9b46_ae6e1de9b801.slice/crio-fd79763f79cd75d5436e4e58e5840c8f4d571b3daabbd9c0bb06d91eba2799c5 WatchSource:0}: Error finding container fd79763f79cd75d5436e4e58e5840c8f4d571b3daabbd9c0bb06d91eba2799c5: Status 404 returned error can't find the container with id fd79763f79cd75d5436e4e58e5840c8f4d571b3daabbd9c0bb06d91eba2799c5 Sep 14 10:05:30.556146 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:30.556100 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" event={"ID":"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801","Type":"ContainerStarted","Data":"6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0"} Sep 14 10:05:30.556573 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:30.556163 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" event={"ID":"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801","Type":"ContainerStarted","Data":"fd79763f79cd75d5436e4e58e5840c8f4d571b3daabbd9c0bb06d91eba2799c5"} Sep 14 10:05:30.556573 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:30.556291 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:05:30.585494 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:30.585443 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" podStartSLOduration=1.585422852 podStartE2EDuration="1.585422852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:05:30.584408811 +0000 UTC m=+1347.279534117" watchObservedRunningTime="2026-09-14 10:05:30.585422852 +0000 UTC m=+1347.280548157" Sep 14 10:05:41.564723 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:05:41.564692 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:06:04.513035 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:04.513005 2550 scope.go:117] "RemoveContainer" containerID="0b34bb31eaed0d035f1fe30d269a1ebce6136f719f8fda82917979307f278c17" Sep 14 10:06:49.062274 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.062240 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m"] Sep 14 10:06:49.062858 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.062496 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" podUID="5c244ab8-5b4b-4100-b40f-032440e6b89d" containerName="maas-api" containerID="cri-o://3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4" gracePeriod=30 Sep 14 10:06:49.226170 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.226130 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" podUID="5c244ab8-5b4b-4100-b40f-032440e6b89d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.60:8443/health\": dial tcp 10.132.0.60:8443: connect: connection refused" Sep 14 10:06:49.316221 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.316165 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:06:49.410141 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.410108 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-metrics-certs\") pod \"5c244ab8-5b4b-4100-b40f-032440e6b89d\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " Sep 14 10:06:49.410305 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.410161 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5wms\" (UniqueName: \"kubernetes.io/projected/5c244ab8-5b4b-4100-b40f-032440e6b89d-kube-api-access-h5wms\") pod \"5c244ab8-5b4b-4100-b40f-032440e6b89d\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " Sep 14 10:06:49.410305 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.410205 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c244ab8-5b4b-4100-b40f-032440e6b89d-openshift-service-ca\") pod \"5c244ab8-5b4b-4100-b40f-032440e6b89d\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " Sep 14 10:06:49.410305 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.410253 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-maas-api-tls\") pod \"5c244ab8-5b4b-4100-b40f-032440e6b89d\" (UID: \"5c244ab8-5b4b-4100-b40f-032440e6b89d\") " Sep 14 10:06:49.410608 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.410575 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c244ab8-5b4b-4100-b40f-032440e6b89d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c244ab8-5b4b-4100-b40f-032440e6b89d" (UID: "5c244ab8-5b4b-4100-b40f-032440e6b89d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:06:49.412282 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.412249 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c244ab8-5b4b-4100-b40f-032440e6b89d" (UID: "5c244ab8-5b4b-4100-b40f-032440e6b89d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:06:49.412391 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.412330 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5c244ab8-5b4b-4100-b40f-032440e6b89d" (UID: "5c244ab8-5b4b-4100-b40f-032440e6b89d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:06:49.412442 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.412408 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c244ab8-5b4b-4100-b40f-032440e6b89d-kube-api-access-h5wms" (OuterVolumeSpecName: "kube-api-access-h5wms") pod "5c244ab8-5b4b-4100-b40f-032440e6b89d" (UID: "5c244ab8-5b4b-4100-b40f-032440e6b89d"). InnerVolumeSpecName "kube-api-access-h5wms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:06:49.511622 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.511592 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:06:49.511622 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.511621 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c244ab8-5b4b-4100-b40f-032440e6b89d-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:06:49.511833 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.511631 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5wms\" (UniqueName: \"kubernetes.io/projected/5c244ab8-5b4b-4100-b40f-032440e6b89d-kube-api-access-h5wms\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:06:49.511833 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.511667 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c244ab8-5b4b-4100-b40f-032440e6b89d-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:06:49.987489 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.987455 2550 generic.go:356] "Generic (PLEG): container finished" podID="5c244ab8-5b4b-4100-b40f-032440e6b89d" containerID="3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4" exitCode=0 Sep 14 10:06:49.987672 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.987505 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" Sep 14 10:06:49.987672 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.987543 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" event={"ID":"5c244ab8-5b4b-4100-b40f-032440e6b89d","Type":"ContainerDied","Data":"3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4"} Sep 14 10:06:49.987672 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.987599 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m" event={"ID":"5c244ab8-5b4b-4100-b40f-032440e6b89d","Type":"ContainerDied","Data":"33a80eac5b9310c5640e58af03deb44f812c04d6c358d24549ea767a168828bd"} Sep 14 10:06:49.987672 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.987626 2550 scope.go:117] "RemoveContainer" containerID="3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4" Sep 14 10:06:49.996667 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.996651 2550 scope.go:117] "RemoveContainer" containerID="3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4" Sep 14 10:06:49.996944 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:06:49.996926 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4\": container with ID starting with 3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4 not found: ID does not exist" containerID="3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4" Sep 14 10:06:49.997014 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:49.996960 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4"} err="failed to get container status \"3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4\": rpc error: code = NotFound desc = could not find container \"3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4\": container with ID starting with 3607c332d69b805693350ff9b42cf87a5c548222a7910595d800f26a48be69c4 not found: ID does not exist" Sep 14 10:06:50.005583 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:50.005561 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m"] Sep 14 10:06:50.007614 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:50.007594 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75017ebc-b8fc595b9-26f8m"] Sep 14 10:06:51.826616 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:06:51.826582 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c244ab8-5b4b-4100-b40f-032440e6b89d" path="/var/lib/kubelet/pods/5c244ab8-5b4b-4100-b40f-032440e6b89d/volumes" Sep 14 10:07:04.535551 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:07:04.535466 2550 scope.go:117] "RemoveContainer" containerID="51c241cb3956c9be31c4cda18d0f47232fa4e9d5a2d9368e05ab7bfaf6afabab" Sep 14 10:08:03.906505 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:03.906480 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:08:03.911048 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:03.911023 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:08:48.264084 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.264041 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r"] Sep 14 10:08:48.266424 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.264774 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c244ab8-5b4b-4100-b40f-032440e6b89d" containerName="maas-api" Sep 14 10:08:48.266424 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.264793 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c244ab8-5b4b-4100-b40f-032440e6b89d" containerName="maas-api" Sep 14 10:08:48.266424 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.264905 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c244ab8-5b4b-4100-b40f-032440e6b89d" containerName="maas-api" Sep 14 10:08:48.269169 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.269146 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.271654 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.271612 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-29794e-serving-cert\"" Sep 14 10:08:48.280473 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.279480 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r"] Sep 14 10:08:48.448977 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.448939 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-maas-api-tls\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.448977 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.448980 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pmp\" (UniqueName: \"kubernetes.io/projected/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-kube-api-access-44pmp\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.449195 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.449014 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-metrics-certs\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.449195 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.449065 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-openshift-service-ca\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.549911 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.549836 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-maas-api-tls\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.549911 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.549877 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pmp\" (UniqueName: \"kubernetes.io/projected/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-kube-api-access-44pmp\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.549911 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.549906 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-metrics-certs\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.550132 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.549945 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-openshift-service-ca\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.550628 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.550605 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-openshift-service-ca\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.552238 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.552218 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-metrics-certs\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.552428 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.552405 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-maas-api-tls\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.558211 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.558187 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pmp\" (UniqueName: \"kubernetes.io/projected/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-kube-api-access-44pmp\") pod \"maas-api-e2e-api-a-29794e-588dcccc4-zzg9r\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.586136 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.586110 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:48.864783 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.864756 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r"] Sep 14 10:08:48.866830 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:08:48.866801 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f621568_0c5e_41ad_bb4b_f4841bdcbc26.slice/crio-d8e6f8f70371662935ce33ccd725e2ba84b07065092a9ef846b36e2c3a93df8c WatchSource:0}: Error finding container d8e6f8f70371662935ce33ccd725e2ba84b07065092a9ef846b36e2c3a93df8c: Status 404 returned error can't find the container with id d8e6f8f70371662935ce33ccd725e2ba84b07065092a9ef846b36e2c3a93df8c Sep 14 10:08:48.868335 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.868319 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 10:08:48.963766 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.963739 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k"] Sep 14 10:08:48.968300 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.968283 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:08:48.975907 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.975890 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-29794e-dockercfg-dgvsh\"" Sep 14 10:08:48.976213 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.976194 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-29794e\"" Sep 14 10:08:48.989308 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:48.989288 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k"] Sep 14 10:08:49.021476 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.021448 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n"] Sep 14 10:08:49.024974 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.024958 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:08:49.065938 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.065911 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n"] Sep 14 10:08:49.157218 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.157141 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkdj\" (UniqueName: \"kubernetes.io/projected/09a69e40-7e68-4e33-8ba7-ca49edde49cc-kube-api-access-4wkdj\") pod \"payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n\" (UID: \"09a69e40-7e68-4e33-8ba7-ca49edde49cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:08:49.157218 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.157186 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721657f5-f642-47b4-b14d-83edb98751fe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k\" (UID: \"721657f5-f642-47b4-b14d-83edb98751fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:08:49.157394 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.157291 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a69e40-7e68-4e33-8ba7-ca49edde49cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n\" (UID: \"09a69e40-7e68-4e33-8ba7-ca49edde49cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:08:49.157394 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.157323 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf7s9\" (UniqueName: \"kubernetes.io/projected/721657f5-f642-47b4-b14d-83edb98751fe-kube-api-access-wf7s9\") pod \"payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k\" (UID: \"721657f5-f642-47b4-b14d-83edb98751fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:08:49.257952 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.257929 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a69e40-7e68-4e33-8ba7-ca49edde49cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n\" (UID: \"09a69e40-7e68-4e33-8ba7-ca49edde49cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:08:49.258049 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.257971 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7s9\" (UniqueName: \"kubernetes.io/projected/721657f5-f642-47b4-b14d-83edb98751fe-kube-api-access-wf7s9\") pod \"payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k\" (UID: \"721657f5-f642-47b4-b14d-83edb98751fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:08:49.258049 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.258009 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkdj\" (UniqueName: \"kubernetes.io/projected/09a69e40-7e68-4e33-8ba7-ca49edde49cc-kube-api-access-4wkdj\") pod \"payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n\" (UID: \"09a69e40-7e68-4e33-8ba7-ca49edde49cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:08:49.258049 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.258028 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721657f5-f642-47b4-b14d-83edb98751fe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k\" (UID: \"721657f5-f642-47b4-b14d-83edb98751fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:08:49.258671 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.258652 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a69e40-7e68-4e33-8ba7-ca49edde49cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n\" (UID: \"09a69e40-7e68-4e33-8ba7-ca49edde49cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:08:49.258718 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.258690 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721657f5-f642-47b4-b14d-83edb98751fe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k\" (UID: \"721657f5-f642-47b4-b14d-83edb98751fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:08:49.266571 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.266545 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkdj\" (UniqueName: \"kubernetes.io/projected/09a69e40-7e68-4e33-8ba7-ca49edde49cc-kube-api-access-4wkdj\") pod \"payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n\" (UID: \"09a69e40-7e68-4e33-8ba7-ca49edde49cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:08:49.266876 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.266863 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7s9\" (UniqueName: \"kubernetes.io/projected/721657f5-f642-47b4-b14d-83edb98751fe-kube-api-access-wf7s9\") pod \"payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k\" (UID: \"721657f5-f642-47b4-b14d-83edb98751fe\") " pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:08:49.278902 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.278882 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:08:49.335443 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.335271 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:08:49.559231 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.559207 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k"] Sep 14 10:08:49.561346 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:08:49.561311 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721657f5_f642_47b4_b14d_83edb98751fe.slice/crio-99cb41eb8dce83ff0653213478b2aefb5de9719e28b11f09161b9a7617b6c676 WatchSource:0}: Error finding container 99cb41eb8dce83ff0653213478b2aefb5de9719e28b11f09161b9a7617b6c676: Status 404 returned error can't find the container with id 99cb41eb8dce83ff0653213478b2aefb5de9719e28b11f09161b9a7617b6c676 Sep 14 10:08:49.616675 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.616625 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n"] Sep 14 10:08:49.617136 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:08:49.617114 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a69e40_7e68_4e33_8ba7_ca49edde49cc.slice/crio-52709dcc09a0dc75cf0d2375445cf7112be27847e3f815fd6467941170873326 WatchSource:0}: Error finding container 52709dcc09a0dc75cf0d2375445cf7112be27847e3f815fd6467941170873326: Status 404 returned error can't find the container with id 52709dcc09a0dc75cf0d2375445cf7112be27847e3f815fd6467941170873326 Sep 14 10:08:49.645109 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.645083 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" event={"ID":"721657f5-f642-47b4-b14d-83edb98751fe","Type":"ContainerStarted","Data":"d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01"} Sep 14 10:08:49.645232 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.645138 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" event={"ID":"721657f5-f642-47b4-b14d-83edb98751fe","Type":"ContainerStarted","Data":"99cb41eb8dce83ff0653213478b2aefb5de9719e28b11f09161b9a7617b6c676"} Sep 14 10:08:49.645232 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.645172 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:08:49.647147 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.647109 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" event={"ID":"9f621568-0c5e-41ad-bb4b-f4841bdcbc26","Type":"ContainerStarted","Data":"789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be"} Sep 14 10:08:49.647246 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.647176 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:08:49.647246 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.647203 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" event={"ID":"9f621568-0c5e-41ad-bb4b-f4841bdcbc26","Type":"ContainerStarted","Data":"d8e6f8f70371662935ce33ccd725e2ba84b07065092a9ef846b36e2c3a93df8c"} Sep 14 10:08:49.648587 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.648558 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" event={"ID":"09a69e40-7e68-4e33-8ba7-ca49edde49cc","Type":"ContainerStarted","Data":"52709dcc09a0dc75cf0d2375445cf7112be27847e3f815fd6467941170873326"} Sep 14 10:08:49.663694 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.663631 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" podStartSLOduration=1.663618151 podStartE2EDuration="1.663618151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:08:49.662457287 +0000 UTC m=+1546.357582817" watchObservedRunningTime="2026-09-14 10:08:49.663618151 +0000 UTC m=+1546.358743456" Sep 14 10:08:49.682099 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:49.682055 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" podStartSLOduration=1.320013215 podStartE2EDuration="1.68204096s" totalImagesPullingTime="362.027745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:08:49.680097981 +0000 UTC m=+1546.375223289" watchObservedRunningTime="2026-09-14 10:08:49.68204096 +0000 UTC m=+1546.377166269" Sep 14 10:08:50.656544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:50.656510 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" event={"ID":"09a69e40-7e68-4e33-8ba7-ca49edde49cc","Type":"ContainerStarted","Data":"90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9"} Sep 14 10:08:50.656986 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:50.656587 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:08:50.678979 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:50.678921 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" podStartSLOduration=2.6789023309999997 podStartE2EDuration="2.678902331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:08:50.673926665 +0000 UTC m=+1547.369051970" watchObservedRunningTime="2026-09-14 10:08:50.678902331 +0000 UTC m=+1547.374027637" Sep 14 10:08:55.662019 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:08:55.661991 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:09:00.659148 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:00.659114 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:09:01.665390 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:01.665361 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:09:18.389509 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.389469 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx"] Sep 14 10:09:18.393357 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.393332 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.395897 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.395873 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-03fd6d-serving-cert\"" Sep 14 10:09:18.405628 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.405605 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx"] Sep 14 10:09:18.539294 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.539257 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4nm\" (UniqueName: \"kubernetes.io/projected/ece13ac0-38b0-408e-a5a0-a4309aadd56f-kube-api-access-fh4nm\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.539294 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.539292 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece13ac0-38b0-408e-a5a0-a4309aadd56f-openshift-service-ca\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.539499 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.539313 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-metrics-certs\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.539499 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.539493 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-maas-api-tls\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.641119 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.641024 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-maas-api-tls\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.641119 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.641085 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4nm\" (UniqueName: \"kubernetes.io/projected/ece13ac0-38b0-408e-a5a0-a4309aadd56f-kube-api-access-fh4nm\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.641119 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.641116 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece13ac0-38b0-408e-a5a0-a4309aadd56f-openshift-service-ca\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.641370 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.641143 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-metrics-certs\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.641370 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:09:18.641183 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-serving-cert: secret "maas-api-e2e-api-b-03fd6d-serving-cert" not found Sep 14 10:09:18.641370 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:09:18.641243 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-maas-api-tls podName:ece13ac0-38b0-408e-a5a0-a4309aadd56f nodeName:}" failed. No retries permitted until 2026-09-14 10:09:19.141226481 +0000 UTC m=+1575.836351764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-maas-api-tls") pod "maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" (UID: "ece13ac0-38b0-408e-a5a0-a4309aadd56f") : secret "maas-api-e2e-api-b-03fd6d-serving-cert" not found Sep 14 10:09:18.641867 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.641845 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece13ac0-38b0-408e-a5a0-a4309aadd56f-openshift-service-ca\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.643814 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.643794 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-metrics-certs\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:18.652660 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:18.652611 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4nm\" (UniqueName: \"kubernetes.io/projected/ece13ac0-38b0-408e-a5a0-a4309aadd56f-kube-api-access-fh4nm\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:19.146369 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:19.146329 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-maas-api-tls\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:19.148692 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:19.148667 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-maas-api-tls\") pod \"maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:19.306185 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:19.306150 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:19.582466 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:19.582442 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx"] Sep 14 10:09:19.584785 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:09:19.584761 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece13ac0_38b0_408e_a5a0_a4309aadd56f.slice/crio-9f89a7f5ae8bdc962b9024bc164a2786f8cb8dac0da76e412f9efbe06aafdb9d WatchSource:0}: Error finding container 9f89a7f5ae8bdc962b9024bc164a2786f8cb8dac0da76e412f9efbe06aafdb9d: Status 404 returned error can't find the container with id 9f89a7f5ae8bdc962b9024bc164a2786f8cb8dac0da76e412f9efbe06aafdb9d Sep 14 10:09:19.826467 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:19.826424 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" event={"ID":"ece13ac0-38b0-408e-a5a0-a4309aadd56f","Type":"ContainerStarted","Data":"9f89a7f5ae8bdc962b9024bc164a2786f8cb8dac0da76e412f9efbe06aafdb9d"} Sep 14 10:09:20.828865 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:20.828827 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" event={"ID":"ece13ac0-38b0-408e-a5a0-a4309aadd56f","Type":"ContainerStarted","Data":"a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61"} Sep 14 10:09:20.829244 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:20.828951 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:20.850796 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:20.850747 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" podStartSLOduration=2.497350986 podStartE2EDuration="2.850728986s" totalImagesPullingTime="353.378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:09:20.846349535 +0000 UTC m=+1577.541474843" watchObservedRunningTime="2026-09-14 10:09:20.850728986 +0000 UTC m=+1577.545854293" Sep 14 10:09:26.839824 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:26.839793 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:09:59.613514 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:59.613429 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r"] Sep 14 10:09:59.614114 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:59.613697 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" podUID="9f621568-0c5e-41ad-bb4b-f4841bdcbc26" containerName="maas-api" containerID="cri-o://789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be" gracePeriod=30 Sep 14 10:09:59.670810 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:59.670777 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k"] Sep 14 10:09:59.671345 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:59.671006 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" podUID="721657f5-f642-47b4-b14d-83edb98751fe" containerName="payload-processing" containerID="cri-o://d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01" gracePeriod=30 Sep 14 10:09:59.679840 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:59.679813 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n"] Sep 14 10:09:59.681145 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:59.680328 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" podUID="09a69e40-7e68-4e33-8ba7-ca49edde49cc" containerName="payload-pre-processing" containerID="cri-o://90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9" gracePeriod=30 Sep 14 10:09:59.962068 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:09:59.962038 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:10:00.005613 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.005589 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:10:00.009246 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.009215 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:10:00.048208 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.048174 2550 generic.go:356] "Generic (PLEG): container finished" podID="721657f5-f642-47b4-b14d-83edb98751fe" containerID="d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01" exitCode=0 Sep 14 10:10:00.048393 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.048236 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" Sep 14 10:10:00.048393 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.048278 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" event={"ID":"721657f5-f642-47b4-b14d-83edb98751fe","Type":"ContainerDied","Data":"d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01"} Sep 14 10:10:00.048393 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.048330 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k" event={"ID":"721657f5-f642-47b4-b14d-83edb98751fe","Type":"ContainerDied","Data":"99cb41eb8dce83ff0653213478b2aefb5de9719e28b11f09161b9a7617b6c676"} Sep 14 10:10:00.048393 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.048360 2550 scope.go:117] "RemoveContainer" containerID="d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01" Sep 14 10:10:00.050000 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.049976 2550 generic.go:356] "Generic (PLEG): container finished" podID="9f621568-0c5e-41ad-bb4b-f4841bdcbc26" containerID="789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be" exitCode=0 Sep 14 10:10:00.050125 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.050029 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" Sep 14 10:10:00.050200 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.050129 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" event={"ID":"9f621568-0c5e-41ad-bb4b-f4841bdcbc26","Type":"ContainerDied","Data":"789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be"} Sep 14 10:10:00.050200 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.050173 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r" event={"ID":"9f621568-0c5e-41ad-bb4b-f4841bdcbc26","Type":"ContainerDied","Data":"d8e6f8f70371662935ce33ccd725e2ba84b07065092a9ef846b36e2c3a93df8c"} Sep 14 10:10:00.051843 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.051691 2550 generic.go:356] "Generic (PLEG): container finished" podID="09a69e40-7e68-4e33-8ba7-ca49edde49cc" containerID="90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9" exitCode=0 Sep 14 10:10:00.051843 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.051746 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" Sep 14 10:10:00.051843 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.051745 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" event={"ID":"09a69e40-7e68-4e33-8ba7-ca49edde49cc","Type":"ContainerDied","Data":"90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9"} Sep 14 10:10:00.051843 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.051795 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n" event={"ID":"09a69e40-7e68-4e33-8ba7-ca49edde49cc","Type":"ContainerDied","Data":"52709dcc09a0dc75cf0d2375445cf7112be27847e3f815fd6467941170873326"} Sep 14 10:10:00.056269 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.056243 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-metrics-certs\") pod \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " Sep 14 10:10:00.056386 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.056341 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44pmp\" (UniqueName: \"kubernetes.io/projected/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-kube-api-access-44pmp\") pod \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " Sep 14 10:10:00.056449 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.056393 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-maas-api-tls\") pod \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " Sep 14 10:10:00.056507 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.056447 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-openshift-service-ca\") pod \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\" (UID: \"9f621568-0c5e-41ad-bb4b-f4841bdcbc26\") " Sep 14 10:10:00.057038 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.057011 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f621568-0c5e-41ad-bb4b-f4841bdcbc26" (UID: "9f621568-0c5e-41ad-bb4b-f4841bdcbc26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:00.058663 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.058613 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9f621568-0c5e-41ad-bb4b-f4841bdcbc26" (UID: "9f621568-0c5e-41ad-bb4b-f4841bdcbc26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:10:00.058780 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.058616 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-kube-api-access-44pmp" (OuterVolumeSpecName: "kube-api-access-44pmp") pod "9f621568-0c5e-41ad-bb4b-f4841bdcbc26" (UID: "9f621568-0c5e-41ad-bb4b-f4841bdcbc26"). InnerVolumeSpecName "kube-api-access-44pmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:00.058900 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.058816 2550 scope.go:117] "RemoveContainer" containerID="d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01" Sep 14 10:10:00.059016 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.058984 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f621568-0c5e-41ad-bb4b-f4841bdcbc26" (UID: "9f621568-0c5e-41ad-bb4b-f4841bdcbc26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:10:00.059122 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:10:00.059105 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01\": container with ID starting with d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01 not found: ID does not exist" containerID="d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01" Sep 14 10:10:00.059203 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.059130 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01"} err="failed to get container status \"d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01\": rpc error: code = NotFound desc = could not find container \"d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01\": container with ID starting with d2b71d40e3848683e6e96c2d901e211f0182534b9d8060a603b25cec9b53de01 not found: ID does not exist" Sep 14 10:10:00.059203 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.059146 2550 scope.go:117] "RemoveContainer" containerID="789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be" Sep 14 10:10:00.067854 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.067776 2550 scope.go:117] "RemoveContainer" containerID="789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be" Sep 14 10:10:00.068097 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:10:00.068078 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be\": container with ID starting with 789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be not found: ID does not exist" containerID="789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be" Sep 14 10:10:00.068193 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.068103 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be"} err="failed to get container status \"789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be\": rpc error: code = NotFound desc = could not find container \"789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be\": container with ID starting with 789db71754b73564c44251a3c6349ed4b361e5f466092ce6904a5a860c5833be not found: ID does not exist" Sep 14 10:10:00.068193 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.068124 2550 scope.go:117] "RemoveContainer" containerID="90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9" Sep 14 10:10:00.076674 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.076629 2550 scope.go:117] "RemoveContainer" containerID="90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9" Sep 14 10:10:00.076939 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:10:00.076920 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9\": container with ID starting with 90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9 not found: ID does not exist" containerID="90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9" Sep 14 10:10:00.076983 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.076944 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9"} err="failed to get container status \"90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9\": rpc error: code = NotFound desc = could not find container \"90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9\": container with ID starting with 90d270c6a9e065f6e955ac7217b1ff89613f241324da099b6f10e5762ae445a9 not found: ID does not exist" Sep 14 10:10:00.157969 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.157938 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a69e40-7e68-4e33-8ba7-ca49edde49cc-plugins-config-volume\") pod \"09a69e40-7e68-4e33-8ba7-ca49edde49cc\" (UID: \"09a69e40-7e68-4e33-8ba7-ca49edde49cc\") " Sep 14 10:10:00.158122 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.158026 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721657f5-f642-47b4-b14d-83edb98751fe-plugins-config-volume\") pod \"721657f5-f642-47b4-b14d-83edb98751fe\" (UID: \"721657f5-f642-47b4-b14d-83edb98751fe\") " Sep 14 10:10:00.158122 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.158056 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wkdj\" (UniqueName: \"kubernetes.io/projected/09a69e40-7e68-4e33-8ba7-ca49edde49cc-kube-api-access-4wkdj\") pod \"09a69e40-7e68-4e33-8ba7-ca49edde49cc\" (UID: \"09a69e40-7e68-4e33-8ba7-ca49edde49cc\") " Sep 14 10:10:00.158122 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.158099 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf7s9\" (UniqueName: \"kubernetes.io/projected/721657f5-f642-47b4-b14d-83edb98751fe-kube-api-access-wf7s9\") pod \"721657f5-f642-47b4-b14d-83edb98751fe\" (UID: \"721657f5-f642-47b4-b14d-83edb98751fe\") " Sep 14 10:10:00.158426 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.158391 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a69e40-7e68-4e33-8ba7-ca49edde49cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09a69e40-7e68-4e33-8ba7-ca49edde49cc" (UID: "09a69e40-7e68-4e33-8ba7-ca49edde49cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:00.158601 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.158407 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:00.158601 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.158472 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:00.158601 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.158435 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721657f5-f642-47b4-b14d-83edb98751fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "721657f5-f642-47b4-b14d-83edb98751fe" (UID: "721657f5-f642-47b4-b14d-83edb98751fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:00.158601 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.158489 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:00.158601 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.158532 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44pmp\" (UniqueName: \"kubernetes.io/projected/9f621568-0c5e-41ad-bb4b-f4841bdcbc26-kube-api-access-44pmp\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:00.160388 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.160369 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721657f5-f642-47b4-b14d-83edb98751fe-kube-api-access-wf7s9" (OuterVolumeSpecName: "kube-api-access-wf7s9") pod "721657f5-f642-47b4-b14d-83edb98751fe" (UID: "721657f5-f642-47b4-b14d-83edb98751fe"). InnerVolumeSpecName "kube-api-access-wf7s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:00.160894 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.160878 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a69e40-7e68-4e33-8ba7-ca49edde49cc-kube-api-access-4wkdj" (OuterVolumeSpecName: "kube-api-access-4wkdj") pod "09a69e40-7e68-4e33-8ba7-ca49edde49cc" (UID: "09a69e40-7e68-4e33-8ba7-ca49edde49cc"). InnerVolumeSpecName "kube-api-access-4wkdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:00.259627 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.259565 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf7s9\" (UniqueName: \"kubernetes.io/projected/721657f5-f642-47b4-b14d-83edb98751fe-kube-api-access-wf7s9\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:00.259627 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.259590 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a69e40-7e68-4e33-8ba7-ca49edde49cc-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:00.259627 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.259599 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721657f5-f642-47b4-b14d-83edb98751fe-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:00.259627 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.259608 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wkdj\" (UniqueName: \"kubernetes.io/projected/09a69e40-7e68-4e33-8ba7-ca49edde49cc-kube-api-access-4wkdj\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:00.376654 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.376610 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k"] Sep 14 10:10:00.382432 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.382409 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29794e-5b84584b7f-tv58k"] Sep 14 10:10:00.393770 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.393745 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r"] Sep 14 10:10:00.401558 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.401536 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-29794e-588dcccc4-zzg9r"] Sep 14 10:10:00.413124 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.413097 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n"] Sep 14 10:10:00.418630 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:00.418588 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29794e-69bb6d7bb6-9zp6n"] Sep 14 10:10:01.832299 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:01.832264 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09a69e40-7e68-4e33-8ba7-ca49edde49cc" path="/var/lib/kubelet/pods/09a69e40-7e68-4e33-8ba7-ca49edde49cc/volumes" Sep 14 10:10:01.832669 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:01.832626 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="721657f5-f642-47b4-b14d-83edb98751fe" path="/var/lib/kubelet/pods/721657f5-f642-47b4-b14d-83edb98751fe/volumes" Sep 14 10:10:01.832993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:01.832977 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f621568-0c5e-41ad-bb4b-f4841bdcbc26" path="/var/lib/kubelet/pods/9f621568-0c5e-41ad-bb4b-f4841bdcbc26/volumes" Sep 14 10:10:23.745398 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:23.745365 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx"] Sep 14 10:10:23.745877 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:23.745594 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" podUID="ece13ac0-38b0-408e-a5a0-a4309aadd56f" containerName="maas-api" containerID="cri-o://a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61" gracePeriod=30 Sep 14 10:10:24.008990 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.008928 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:10:24.098298 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.098262 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece13ac0-38b0-408e-a5a0-a4309aadd56f-openshift-service-ca\") pod \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " Sep 14 10:10:24.098476 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.098310 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh4nm\" (UniqueName: \"kubernetes.io/projected/ece13ac0-38b0-408e-a5a0-a4309aadd56f-kube-api-access-fh4nm\") pod \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " Sep 14 10:10:24.098476 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.098402 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-maas-api-tls\") pod \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " Sep 14 10:10:24.098603 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.098486 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-metrics-certs\") pod \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\" (UID: \"ece13ac0-38b0-408e-a5a0-a4309aadd56f\") " Sep 14 10:10:24.098724 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.098695 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece13ac0-38b0-408e-a5a0-a4309aadd56f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ece13ac0-38b0-408e-a5a0-a4309aadd56f" (UID: "ece13ac0-38b0-408e-a5a0-a4309aadd56f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:24.098908 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.098889 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece13ac0-38b0-408e-a5a0-a4309aadd56f-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:24.100567 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.100539 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece13ac0-38b0-408e-a5a0-a4309aadd56f-kube-api-access-fh4nm" (OuterVolumeSpecName: "kube-api-access-fh4nm") pod "ece13ac0-38b0-408e-a5a0-a4309aadd56f" (UID: "ece13ac0-38b0-408e-a5a0-a4309aadd56f"). InnerVolumeSpecName "kube-api-access-fh4nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:24.100697 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.100565 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ece13ac0-38b0-408e-a5a0-a4309aadd56f" (UID: "ece13ac0-38b0-408e-a5a0-a4309aadd56f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:10:24.100697 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.100575 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ece13ac0-38b0-408e-a5a0-a4309aadd56f" (UID: "ece13ac0-38b0-408e-a5a0-a4309aadd56f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:10:24.197331 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.197290 2550 generic.go:356] "Generic (PLEG): container finished" podID="ece13ac0-38b0-408e-a5a0-a4309aadd56f" containerID="a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61" exitCode=0 Sep 14 10:10:24.197506 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.197341 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" Sep 14 10:10:24.197506 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.197370 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" event={"ID":"ece13ac0-38b0-408e-a5a0-a4309aadd56f","Type":"ContainerDied","Data":"a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61"} Sep 14 10:10:24.197506 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.197428 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx" event={"ID":"ece13ac0-38b0-408e-a5a0-a4309aadd56f","Type":"ContainerDied","Data":"9f89a7f5ae8bdc962b9024bc164a2786f8cb8dac0da76e412f9efbe06aafdb9d"} Sep 14 10:10:24.197506 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.197451 2550 scope.go:117] "RemoveContainer" containerID="a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61" Sep 14 10:10:24.199430 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.199411 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:24.199533 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.199433 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece13ac0-38b0-408e-a5a0-a4309aadd56f-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:24.199533 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.199448 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh4nm\" (UniqueName: \"kubernetes.io/projected/ece13ac0-38b0-408e-a5a0-a4309aadd56f-kube-api-access-fh4nm\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:24.208620 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.208604 2550 scope.go:117] "RemoveContainer" containerID="a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61" Sep 14 10:10:24.208991 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:10:24.208955 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61\": container with ID starting with a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61 not found: ID does not exist" containerID="a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61" Sep 14 10:10:24.209098 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.209003 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61"} err="failed to get container status \"a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61\": rpc error: code = NotFound desc = could not find container \"a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61\": container with ID starting with a8efe67a954ba1c21cd2561191ddb7ccde0a7e8aeea0e3e9d2e30ed988ad0d61 not found: ID does not exist" Sep 14 10:10:24.221786 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.221764 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx"] Sep 14 10:10:24.223798 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:24.223778 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03fd6d-74955fd4b-wgzmx"] Sep 14 10:10:25.824948 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:25.824906 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece13ac0-38b0-408e-a5a0-a4309aadd56f" path="/var/lib/kubelet/pods/ece13ac0-38b0-408e-a5a0-a4309aadd56f/volumes" Sep 14 10:10:28.897650 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:28.897605 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5"] Sep 14 10:10:28.898062 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:28.897837 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" podUID="2bf07770-24c1-4eaf-a696-de83cc22dcd2" containerName="payload-processing" containerID="cri-o://d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31" gracePeriod=30 Sep 14 10:10:28.907535 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:28.907512 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs"] Sep 14 10:10:28.907786 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:28.907735 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" podUID="ba6b37aa-2afd-49aa-aa1c-0d33d78abb88" containerName="payload-pre-processing" containerID="cri-o://c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3" gracePeriod=30 Sep 14 10:10:29.190928 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.190905 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 10:10:29.212533 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.212507 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 10:10:29.228598 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.228568 2550 generic.go:356] "Generic (PLEG): container finished" podID="2bf07770-24c1-4eaf-a696-de83cc22dcd2" containerID="d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31" exitCode=0 Sep 14 10:10:29.228747 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.228613 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" Sep 14 10:10:29.228747 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.228657 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" event={"ID":"2bf07770-24c1-4eaf-a696-de83cc22dcd2","Type":"ContainerDied","Data":"d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31"} Sep 14 10:10:29.228747 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.228729 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5" event={"ID":"2bf07770-24c1-4eaf-a696-de83cc22dcd2","Type":"ContainerDied","Data":"46bbd5b525cc701643e728ecd466c15c865f570714784bacd8383972122b3b18"} Sep 14 10:10:29.228895 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.228762 2550 scope.go:117] "RemoveContainer" containerID="d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31" Sep 14 10:10:29.230376 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.230342 2550 generic.go:356] "Generic (PLEG): container finished" podID="ba6b37aa-2afd-49aa-aa1c-0d33d78abb88" containerID="c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3" exitCode=0 Sep 14 10:10:29.230495 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.230377 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" event={"ID":"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88","Type":"ContainerDied","Data":"c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3"} Sep 14 10:10:29.230495 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.230396 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" Sep 14 10:10:29.230495 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.230425 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs" event={"ID":"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88","Type":"ContainerDied","Data":"a96fa6435f8a4635da46a15234e91616b75b18a29cd9c8515eefb88275c971d3"} Sep 14 10:10:29.239794 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.239774 2550 scope.go:117] "RemoveContainer" containerID="d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31" Sep 14 10:10:29.240155 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:10:29.240131 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31\": container with ID starting with d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31 not found: ID does not exist" containerID="d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31" Sep 14 10:10:29.240251 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.240166 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31"} err="failed to get container status \"d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31\": rpc error: code = NotFound desc = could not find container \"d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31\": container with ID starting with d2ffe08ad7051be9b81841be3623d393779b7d733e480e99868a590c257c0b31 not found: ID does not exist" Sep 14 10:10:29.240251 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.240186 2550 scope.go:117] "RemoveContainer" containerID="c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3" Sep 14 10:10:29.252592 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.252572 2550 scope.go:117] "RemoveContainer" containerID="c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3" Sep 14 10:10:29.252873 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:10:29.252856 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3\": container with ID starting with c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3 not found: ID does not exist" containerID="c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3" Sep 14 10:10:29.252921 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.252881 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3"} err="failed to get container status \"c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3\": rpc error: code = NotFound desc = could not find container \"c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3\": container with ID starting with c92acd1c1a0056bac12b4becdadd5a29fc3bd6f32a3a9547725f1821226850a3 not found: ID does not exist" Sep 14 10:10:29.351049 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.351017 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k27w\" (UniqueName: \"kubernetes.io/projected/2bf07770-24c1-4eaf-a696-de83cc22dcd2-kube-api-access-7k27w\") pod \"2bf07770-24c1-4eaf-a696-de83cc22dcd2\" (UID: \"2bf07770-24c1-4eaf-a696-de83cc22dcd2\") " Sep 14 10:10:29.351241 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.351069 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5529\" (UniqueName: \"kubernetes.io/projected/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-kube-api-access-r5529\") pod \"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88\" (UID: \"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88\") " Sep 14 10:10:29.351241 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.351092 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-plugins-config-volume\") pod \"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88\" (UID: \"ba6b37aa-2afd-49aa-aa1c-0d33d78abb88\") " Sep 14 10:10:29.351374 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.351242 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf07770-24c1-4eaf-a696-de83cc22dcd2-plugins-config-volume\") pod \"2bf07770-24c1-4eaf-a696-de83cc22dcd2\" (UID: \"2bf07770-24c1-4eaf-a696-de83cc22dcd2\") " Sep 14 10:10:29.351564 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.351528 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba6b37aa-2afd-49aa-aa1c-0d33d78abb88" (UID: "ba6b37aa-2afd-49aa-aa1c-0d33d78abb88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:29.351727 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.351575 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf07770-24c1-4eaf-a696-de83cc22dcd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2bf07770-24c1-4eaf-a696-de83cc22dcd2" (UID: "2bf07770-24c1-4eaf-a696-de83cc22dcd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:29.351803 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.351742 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf07770-24c1-4eaf-a696-de83cc22dcd2-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:29.351803 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.351762 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:29.353248 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.353225 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf07770-24c1-4eaf-a696-de83cc22dcd2-kube-api-access-7k27w" (OuterVolumeSpecName: "kube-api-access-7k27w") pod "2bf07770-24c1-4eaf-a696-de83cc22dcd2" (UID: "2bf07770-24c1-4eaf-a696-de83cc22dcd2"). InnerVolumeSpecName "kube-api-access-7k27w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:29.353415 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.353396 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-kube-api-access-r5529" (OuterVolumeSpecName: "kube-api-access-r5529") pod "ba6b37aa-2afd-49aa-aa1c-0d33d78abb88" (UID: "ba6b37aa-2afd-49aa-aa1c-0d33d78abb88"). InnerVolumeSpecName "kube-api-access-r5529". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:29.452307 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.452227 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k27w\" (UniqueName: \"kubernetes.io/projected/2bf07770-24c1-4eaf-a696-de83cc22dcd2-kube-api-access-7k27w\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:29.452307 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.452254 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5529\" (UniqueName: \"kubernetes.io/projected/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88-kube-api-access-r5529\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:29.551565 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.551532 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5"] Sep 14 10:10:29.555455 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.555431 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6812e-554498cd7f-8kpr5"] Sep 14 10:10:29.565965 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.565943 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs"] Sep 14 10:10:29.573798 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.573777 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6812e-f7cdc8b7d-qlpzs"] Sep 14 10:10:29.828153 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.828062 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bf07770-24c1-4eaf-a696-de83cc22dcd2" path="/var/lib/kubelet/pods/2bf07770-24c1-4eaf-a696-de83cc22dcd2/volumes" Sep 14 10:10:29.828668 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:29.828619 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6b37aa-2afd-49aa-aa1c-0d33d78abb88" path="/var/lib/kubelet/pods/ba6b37aa-2afd-49aa-aa1c-0d33d78abb88/volumes" Sep 14 10:10:43.470315 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.470278 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr"] Sep 14 10:10:43.470885 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.470510 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" podUID="cf406175-8413-4320-9043-0188ffc52141" containerName="maas-api" containerID="cri-o://9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb" gracePeriod=30 Sep 14 10:10:43.531209 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.531168 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42"] Sep 14 10:10:43.531725 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.531653 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" podUID="baba4bae-13e2-43e7-a5af-ad81a475d96b" containerName="payload-processing" containerID="cri-o://c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b" gracePeriod=30 Sep 14 10:10:43.546169 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.546121 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv"] Sep 14 10:10:43.546413 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.546363 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" podUID="1ace602e-4cae-4e21-8b8c-efff41247ee7" containerName="payload-pre-processing" containerID="cri-o://138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c" gracePeriod=30 Sep 14 10:10:43.839922 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.839895 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:10:43.856022 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.855995 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:10:43.865472 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.865450 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:10:43.894882 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.894857 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwkk5\" (UniqueName: \"kubernetes.io/projected/1ace602e-4cae-4e21-8b8c-efff41247ee7-kube-api-access-dwkk5\") pod \"1ace602e-4cae-4e21-8b8c-efff41247ee7\" (UID: \"1ace602e-4cae-4e21-8b8c-efff41247ee7\") " Sep 14 10:10:43.895076 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.894949 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8654\" (UniqueName: \"kubernetes.io/projected/baba4bae-13e2-43e7-a5af-ad81a475d96b-kube-api-access-g8654\") pod \"baba4bae-13e2-43e7-a5af-ad81a475d96b\" (UID: \"baba4bae-13e2-43e7-a5af-ad81a475d96b\") " Sep 14 10:10:43.895076 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.894991 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ace602e-4cae-4e21-8b8c-efff41247ee7-plugins-config-volume\") pod \"1ace602e-4cae-4e21-8b8c-efff41247ee7\" (UID: \"1ace602e-4cae-4e21-8b8c-efff41247ee7\") " Sep 14 10:10:43.895076 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.895031 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baba4bae-13e2-43e7-a5af-ad81a475d96b-plugins-config-volume\") pod \"baba4bae-13e2-43e7-a5af-ad81a475d96b\" (UID: \"baba4bae-13e2-43e7-a5af-ad81a475d96b\") " Sep 14 10:10:43.895076 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.895057 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-metrics-certs\") pod \"cf406175-8413-4320-9043-0188ffc52141\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " Sep 14 10:10:43.895309 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.895112 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-maas-api-tls\") pod \"cf406175-8413-4320-9043-0188ffc52141\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " Sep 14 10:10:43.895309 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.895143 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbj4h\" (UniqueName: \"kubernetes.io/projected/cf406175-8413-4320-9043-0188ffc52141-kube-api-access-kbj4h\") pod \"cf406175-8413-4320-9043-0188ffc52141\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " Sep 14 10:10:43.895309 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.895190 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf406175-8413-4320-9043-0188ffc52141-openshift-service-ca\") pod \"cf406175-8413-4320-9043-0188ffc52141\" (UID: \"cf406175-8413-4320-9043-0188ffc52141\") " Sep 14 10:10:43.895516 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.895460 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ace602e-4cae-4e21-8b8c-efff41247ee7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ace602e-4cae-4e21-8b8c-efff41247ee7" (UID: "1ace602e-4cae-4e21-8b8c-efff41247ee7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:43.895966 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.895568 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baba4bae-13e2-43e7-a5af-ad81a475d96b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "baba4bae-13e2-43e7-a5af-ad81a475d96b" (UID: "baba4bae-13e2-43e7-a5af-ad81a475d96b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:43.895966 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.895594 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ace602e-4cae-4e21-8b8c-efff41247ee7-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:43.896131 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.896012 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf406175-8413-4320-9043-0188ffc52141-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf406175-8413-4320-9043-0188ffc52141" (UID: "cf406175-8413-4320-9043-0188ffc52141"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:43.897554 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.897527 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ace602e-4cae-4e21-8b8c-efff41247ee7-kube-api-access-dwkk5" (OuterVolumeSpecName: "kube-api-access-dwkk5") pod "1ace602e-4cae-4e21-8b8c-efff41247ee7" (UID: "1ace602e-4cae-4e21-8b8c-efff41247ee7"). InnerVolumeSpecName "kube-api-access-dwkk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:43.897820 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.897790 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baba4bae-13e2-43e7-a5af-ad81a475d96b-kube-api-access-g8654" (OuterVolumeSpecName: "kube-api-access-g8654") pod "baba4bae-13e2-43e7-a5af-ad81a475d96b" (UID: "baba4bae-13e2-43e7-a5af-ad81a475d96b"). InnerVolumeSpecName "kube-api-access-g8654". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:43.897918 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.897895 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf406175-8413-4320-9043-0188ffc52141-kube-api-access-kbj4h" (OuterVolumeSpecName: "kube-api-access-kbj4h") pod "cf406175-8413-4320-9043-0188ffc52141" (UID: "cf406175-8413-4320-9043-0188ffc52141"). InnerVolumeSpecName "kube-api-access-kbj4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:43.897991 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.897931 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf406175-8413-4320-9043-0188ffc52141" (UID: "cf406175-8413-4320-9043-0188ffc52141"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:10:43.898051 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.898029 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cf406175-8413-4320-9043-0188ffc52141" (UID: "cf406175-8413-4320-9043-0188ffc52141"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:10:43.996791 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.996708 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwkk5\" (UniqueName: \"kubernetes.io/projected/1ace602e-4cae-4e21-8b8c-efff41247ee7-kube-api-access-dwkk5\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:43.996791 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.996736 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8654\" (UniqueName: \"kubernetes.io/projected/baba4bae-13e2-43e7-a5af-ad81a475d96b-kube-api-access-g8654\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:43.996791 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.996746 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baba4bae-13e2-43e7-a5af-ad81a475d96b-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:43.996791 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.996755 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:43.996791 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.996764 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf406175-8413-4320-9043-0188ffc52141-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:43.996791 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.996771 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbj4h\" (UniqueName: \"kubernetes.io/projected/cf406175-8413-4320-9043-0188ffc52141-kube-api-access-kbj4h\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:43.996791 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:43.996780 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf406175-8413-4320-9043-0188ffc52141-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:44.033957 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.033924 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k"] Sep 14 10:10:44.034169 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.034141 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" podUID="fc763f21-0c0e-4ae8-bc3b-ef835738f829" containerName="payload-processing" containerID="cri-o://7a4b802fdbcd6f7f0d62df4e189f5123b9cef7c2086a9ba373c63dcd8483e7f3" gracePeriod=30 Sep 14 10:10:44.049561 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.049531 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr"] Sep 14 10:10:44.049804 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.049778 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" podUID="b5dba100-838a-49dc-bdee-9a0883e98396" containerName="payload-pre-processing" containerID="cri-o://45db35ea48a476f2260fa83b751ab86e3a257e8b3e0dda451f67666fbda37d3e" gracePeriod=30 Sep 14 10:10:44.315302 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.315271 2550 generic.go:356] "Generic (PLEG): container finished" podID="fc763f21-0c0e-4ae8-bc3b-ef835738f829" containerID="7a4b802fdbcd6f7f0d62df4e189f5123b9cef7c2086a9ba373c63dcd8483e7f3" exitCode=0 Sep 14 10:10:44.315384 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.315316 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" event={"ID":"fc763f21-0c0e-4ae8-bc3b-ef835738f829","Type":"ContainerDied","Data":"7a4b802fdbcd6f7f0d62df4e189f5123b9cef7c2086a9ba373c63dcd8483e7f3"} Sep 14 10:10:44.316770 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.316745 2550 generic.go:356] "Generic (PLEG): container finished" podID="cf406175-8413-4320-9043-0188ffc52141" containerID="9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb" exitCode=0 Sep 14 10:10:44.316905 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.316814 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" Sep 14 10:10:44.316905 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.316835 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" event={"ID":"cf406175-8413-4320-9043-0188ffc52141","Type":"ContainerDied","Data":"9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb"} Sep 14 10:10:44.316905 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.316889 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr" event={"ID":"cf406175-8413-4320-9043-0188ffc52141","Type":"ContainerDied","Data":"e8007896ba77f22e48a8cf81eca8911151168d597f662cdaf57b9a02c55d1294"} Sep 14 10:10:44.317079 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.316918 2550 scope.go:117] "RemoveContainer" containerID="9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb" Sep 14 10:10:44.317436 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.317406 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 10:10:44.319374 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.319351 2550 generic.go:356] "Generic (PLEG): container finished" podID="baba4bae-13e2-43e7-a5af-ad81a475d96b" containerID="c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b" exitCode=0 Sep 14 10:10:44.319774 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.319532 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" event={"ID":"baba4bae-13e2-43e7-a5af-ad81a475d96b","Type":"ContainerDied","Data":"c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b"} Sep 14 10:10:44.319891 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.319796 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" event={"ID":"baba4bae-13e2-43e7-a5af-ad81a475d96b","Type":"ContainerDied","Data":"18135673a745f4ce5ca1f12fa51579efa123bcaa5a1b0cb4ca185d94aa330d9e"} Sep 14 10:10:44.319891 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.319566 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42" Sep 14 10:10:44.321597 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.321578 2550 generic.go:356] "Generic (PLEG): container finished" podID="1ace602e-4cae-4e21-8b8c-efff41247ee7" containerID="138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c" exitCode=0 Sep 14 10:10:44.321712 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.321685 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" event={"ID":"1ace602e-4cae-4e21-8b8c-efff41247ee7","Type":"ContainerDied","Data":"138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c"} Sep 14 10:10:44.321789 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.321662 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" Sep 14 10:10:44.321789 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.321747 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv" event={"ID":"1ace602e-4cae-4e21-8b8c-efff41247ee7","Type":"ContainerDied","Data":"18b21a2bd591e265953c19061ef7df0640f1a95c29c6479f240c57229c8c6399"} Sep 14 10:10:44.323614 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.323573 2550 generic.go:356] "Generic (PLEG): container finished" podID="b5dba100-838a-49dc-bdee-9a0883e98396" containerID="45db35ea48a476f2260fa83b751ab86e3a257e8b3e0dda451f67666fbda37d3e" exitCode=0 Sep 14 10:10:44.323614 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.323611 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" event={"ID":"b5dba100-838a-49dc-bdee-9a0883e98396","Type":"ContainerDied","Data":"45db35ea48a476f2260fa83b751ab86e3a257e8b3e0dda451f67666fbda37d3e"} Sep 14 10:10:44.323792 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.323706 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 10:10:44.330462 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.330447 2550 scope.go:117] "RemoveContainer" containerID="9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb" Sep 14 10:10:44.330930 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:10:44.330903 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb\": container with ID starting with 9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb not found: ID does not exist" containerID="9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb" Sep 14 10:10:44.331031 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.330934 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb"} err="failed to get container status \"9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb\": rpc error: code = NotFound desc = could not find container \"9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb\": container with ID starting with 9bb5d9658cf40c7b3f5a21d4c34fc93c47521039edc781803f739c1e57df29fb not found: ID does not exist" Sep 14 10:10:44.331031 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.330955 2550 scope.go:117] "RemoveContainer" containerID="c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b" Sep 14 10:10:44.340051 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.340034 2550 scope.go:117] "RemoveContainer" containerID="c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b" Sep 14 10:10:44.340325 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:10:44.340307 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b\": container with ID starting with c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b not found: ID does not exist" containerID="c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b" Sep 14 10:10:44.340366 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.340332 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b"} err="failed to get container status \"c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b\": rpc error: code = NotFound desc = could not find container \"c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b\": container with ID starting with c7f86c4fe641bf3df820f82b104f2a747ac758bdbd4aad46b625abe3c9c5a26b not found: ID does not exist" Sep 14 10:10:44.340366 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.340348 2550 scope.go:117] "RemoveContainer" containerID="138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c" Sep 14 10:10:44.348121 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.348102 2550 scope.go:117] "RemoveContainer" containerID="138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c" Sep 14 10:10:44.348377 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:10:44.348359 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c\": container with ID starting with 138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c not found: ID does not exist" containerID="138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c" Sep 14 10:10:44.348423 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.348385 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c"} err="failed to get container status \"138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c\": rpc error: code = NotFound desc = could not find container \"138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c\": container with ID starting with 138db08793f6a4968a94f5000cab0f89d8161c00645f2e1bf19567dc07fefd6c not found: ID does not exist" Sep 14 10:10:44.364098 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.364073 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv"] Sep 14 10:10:44.373745 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.373715 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8fd006-69bbb6bdc8-6mqtv"] Sep 14 10:10:44.391244 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.391215 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr"] Sep 14 10:10:44.398250 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.398226 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8fd006-7dbd6c6d77-t56sr"] Sep 14 10:10:44.400506 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.400487 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44djp\" (UniqueName: \"kubernetes.io/projected/b5dba100-838a-49dc-bdee-9a0883e98396-kube-api-access-44djp\") pod \"b5dba100-838a-49dc-bdee-9a0883e98396\" (UID: \"b5dba100-838a-49dc-bdee-9a0883e98396\") " Sep 14 10:10:44.400571 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.400559 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8sd9\" (UniqueName: \"kubernetes.io/projected/fc763f21-0c0e-4ae8-bc3b-ef835738f829-kube-api-access-f8sd9\") pod \"fc763f21-0c0e-4ae8-bc3b-ef835738f829\" (UID: \"fc763f21-0c0e-4ae8-bc3b-ef835738f829\") " Sep 14 10:10:44.400682 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.400670 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5dba100-838a-49dc-bdee-9a0883e98396-plugins-config-volume\") pod \"b5dba100-838a-49dc-bdee-9a0883e98396\" (UID: \"b5dba100-838a-49dc-bdee-9a0883e98396\") " Sep 14 10:10:44.400718 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.400699 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc763f21-0c0e-4ae8-bc3b-ef835738f829-plugins-config-volume\") pod \"fc763f21-0c0e-4ae8-bc3b-ef835738f829\" (UID: \"fc763f21-0c0e-4ae8-bc3b-ef835738f829\") " Sep 14 10:10:44.401082 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.401052 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5dba100-838a-49dc-bdee-9a0883e98396-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5dba100-838a-49dc-bdee-9a0883e98396" (UID: "b5dba100-838a-49dc-bdee-9a0883e98396"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:44.401082 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.401068 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc763f21-0c0e-4ae8-bc3b-ef835738f829-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc763f21-0c0e-4ae8-bc3b-ef835738f829" (UID: "fc763f21-0c0e-4ae8-bc3b-ef835738f829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:10:44.402558 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.402537 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc763f21-0c0e-4ae8-bc3b-ef835738f829-kube-api-access-f8sd9" (OuterVolumeSpecName: "kube-api-access-f8sd9") pod "fc763f21-0c0e-4ae8-bc3b-ef835738f829" (UID: "fc763f21-0c0e-4ae8-bc3b-ef835738f829"). InnerVolumeSpecName "kube-api-access-f8sd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:44.402666 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.402566 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5dba100-838a-49dc-bdee-9a0883e98396-kube-api-access-44djp" (OuterVolumeSpecName: "kube-api-access-44djp") pod "b5dba100-838a-49dc-bdee-9a0883e98396" (UID: "b5dba100-838a-49dc-bdee-9a0883e98396"). InnerVolumeSpecName "kube-api-access-44djp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:10:44.473558 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.473522 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42"] Sep 14 10:10:44.482619 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.482592 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8fd006-79fc74bcfc-zbc42"] Sep 14 10:10:44.502206 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.502181 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44djp\" (UniqueName: \"kubernetes.io/projected/b5dba100-838a-49dc-bdee-9a0883e98396-kube-api-access-44djp\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:44.502314 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.502205 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8sd9\" (UniqueName: \"kubernetes.io/projected/fc763f21-0c0e-4ae8-bc3b-ef835738f829-kube-api-access-f8sd9\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:44.502314 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.502222 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5dba100-838a-49dc-bdee-9a0883e98396-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:44.502314 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:44.502232 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc763f21-0c0e-4ae8-bc3b-ef835738f829-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:10:45.334188 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.334156 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" Sep 14 10:10:45.334188 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.334174 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr" event={"ID":"b5dba100-838a-49dc-bdee-9a0883e98396","Type":"ContainerDied","Data":"5560d6c3a02813dfa619b1d9d3c997c415a34c395ffc6d4bb76643c814512201"} Sep 14 10:10:45.334470 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.334240 2550 scope.go:117] "RemoveContainer" containerID="45db35ea48a476f2260fa83b751ab86e3a257e8b3e0dda451f67666fbda37d3e" Sep 14 10:10:45.335623 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.335542 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" Sep 14 10:10:45.335623 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.335547 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k" event={"ID":"fc763f21-0c0e-4ae8-bc3b-ef835738f829","Type":"ContainerDied","Data":"9911e6140c56c1d257e46cf691d33e5947b9357674964a354226166bc6e9e851"} Sep 14 10:10:45.344428 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.344300 2550 scope.go:117] "RemoveContainer" containerID="7a4b802fdbcd6f7f0d62df4e189f5123b9cef7c2086a9ba373c63dcd8483e7f3" Sep 14 10:10:45.359971 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.359945 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr"] Sep 14 10:10:45.363756 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.363735 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2901ec-5d847dc6cd-nrttr"] Sep 14 10:10:45.374028 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.374006 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k"] Sep 14 10:10:45.379263 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.379242 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2901ec-5877ff9cf9-c2r2k"] Sep 14 10:10:45.824766 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.824736 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ace602e-4cae-4e21-8b8c-efff41247ee7" path="/var/lib/kubelet/pods/1ace602e-4cae-4e21-8b8c-efff41247ee7/volumes" Sep 14 10:10:45.825138 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.825093 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5dba100-838a-49dc-bdee-9a0883e98396" path="/var/lib/kubelet/pods/b5dba100-838a-49dc-bdee-9a0883e98396/volumes" Sep 14 10:10:45.825431 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.825417 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baba4bae-13e2-43e7-a5af-ad81a475d96b" path="/var/lib/kubelet/pods/baba4bae-13e2-43e7-a5af-ad81a475d96b/volumes" Sep 14 10:10:45.825793 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.825780 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf406175-8413-4320-9043-0188ffc52141" path="/var/lib/kubelet/pods/cf406175-8413-4320-9043-0188ffc52141/volumes" Sep 14 10:10:45.826096 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:10:45.826084 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc763f21-0c0e-4ae8-bc3b-ef835738f829" path="/var/lib/kubelet/pods/fc763f21-0c0e-4ae8-bc3b-ef835738f829/volumes" Sep 14 10:11:01.705575 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:01.705539 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2"] Sep 14 10:11:01.706033 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:01.705791 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" podUID="82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4" containerName="payload-processing" containerID="cri-o://52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9" gracePeriod=30 Sep 14 10:11:01.711799 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:01.711768 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf"] Sep 14 10:11:01.712037 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:01.711987 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" podUID="3badb610-7f4b-4f50-a8d7-8c54f955ee1b" containerName="payload-pre-processing" containerID="cri-o://09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec" gracePeriod=30 Sep 14 10:11:01.989066 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:01.989042 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:11:01.991840 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:01.991821 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:11:02.061412 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.061337 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwgdg\" (UniqueName: \"kubernetes.io/projected/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-kube-api-access-gwgdg\") pod \"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4\" (UID: \"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4\") " Sep 14 10:11:02.061655 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.061441 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8kvq\" (UniqueName: \"kubernetes.io/projected/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-kube-api-access-n8kvq\") pod \"3badb610-7f4b-4f50-a8d7-8c54f955ee1b\" (UID: \"3badb610-7f4b-4f50-a8d7-8c54f955ee1b\") " Sep 14 10:11:02.061655 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.061492 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-plugins-config-volume\") pod \"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4\" (UID: \"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4\") " Sep 14 10:11:02.061655 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.061533 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-plugins-config-volume\") pod \"3badb610-7f4b-4f50-a8d7-8c54f955ee1b\" (UID: \"3badb610-7f4b-4f50-a8d7-8c54f955ee1b\") " Sep 14 10:11:02.061983 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.061945 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4" (UID: "82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:11:02.062317 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.062281 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3badb610-7f4b-4f50-a8d7-8c54f955ee1b" (UID: "3badb610-7f4b-4f50-a8d7-8c54f955ee1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:11:02.063585 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.063561 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-kube-api-access-gwgdg" (OuterVolumeSpecName: "kube-api-access-gwgdg") pod "82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4" (UID: "82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4"). InnerVolumeSpecName "kube-api-access-gwgdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:11:02.063732 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.063715 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-kube-api-access-n8kvq" (OuterVolumeSpecName: "kube-api-access-n8kvq") pod "3badb610-7f4b-4f50-a8d7-8c54f955ee1b" (UID: "3badb610-7f4b-4f50-a8d7-8c54f955ee1b"). InnerVolumeSpecName "kube-api-access-n8kvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:11:02.162928 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.162898 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwgdg\" (UniqueName: \"kubernetes.io/projected/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-kube-api-access-gwgdg\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:11:02.162928 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.162924 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8kvq\" (UniqueName: \"kubernetes.io/projected/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-kube-api-access-n8kvq\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:11:02.162928 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.162933 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:11:02.163131 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.162942 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3badb610-7f4b-4f50-a8d7-8c54f955ee1b-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:11:02.431457 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.431416 2550 generic.go:356] "Generic (PLEG): container finished" podID="82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4" containerID="52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9" exitCode=0 Sep 14 10:11:02.431624 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.431462 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" Sep 14 10:11:02.431624 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.431504 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" event={"ID":"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4","Type":"ContainerDied","Data":"52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9"} Sep 14 10:11:02.431624 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.431570 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2" event={"ID":"82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4","Type":"ContainerDied","Data":"787caaadd0ee884f0c5a978b6e07b66993c6dbfc37648f23925ba7a8eff54f90"} Sep 14 10:11:02.431624 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.431599 2550 scope.go:117] "RemoveContainer" containerID="52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9" Sep 14 10:11:02.432994 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.432968 2550 generic.go:356] "Generic (PLEG): container finished" podID="3badb610-7f4b-4f50-a8d7-8c54f955ee1b" containerID="09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec" exitCode=0 Sep 14 10:11:02.433117 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.432999 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" event={"ID":"3badb610-7f4b-4f50-a8d7-8c54f955ee1b","Type":"ContainerDied","Data":"09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec"} Sep 14 10:11:02.433117 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.433016 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" Sep 14 10:11:02.433117 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.433037 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf" event={"ID":"3badb610-7f4b-4f50-a8d7-8c54f955ee1b","Type":"ContainerDied","Data":"99260b9cf5ab97d6ec192b8b0446fab0534a6eb7f96cf77f6a892d74e13f2aba"} Sep 14 10:11:02.446278 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.446259 2550 scope.go:117] "RemoveContainer" containerID="52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9" Sep 14 10:11:02.446612 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:11:02.446588 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9\": container with ID starting with 52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9 not found: ID does not exist" containerID="52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9" Sep 14 10:11:02.446698 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.446622 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9"} err="failed to get container status \"52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9\": rpc error: code = NotFound desc = could not find container \"52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9\": container with ID starting with 52f92f5c880849094a748f7d1528a0ffa113e8ea74b7f0addae9663e13419de9 not found: ID does not exist" Sep 14 10:11:02.446698 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.446665 2550 scope.go:117] "RemoveContainer" containerID="09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec" Sep 14 10:11:02.454983 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.454963 2550 scope.go:117] "RemoveContainer" containerID="09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec" Sep 14 10:11:02.455270 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:11:02.455250 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec\": container with ID starting with 09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec not found: ID does not exist" containerID="09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec" Sep 14 10:11:02.455318 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.455275 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec"} err="failed to get container status \"09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec\": rpc error: code = NotFound desc = could not find container \"09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec\": container with ID starting with 09bd528939a96273970ecc0559dec016919e96b0dd2c46f90c4b45663bcdf8ec not found: ID does not exist" Sep 14 10:11:02.463769 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.463747 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2"] Sep 14 10:11:02.467611 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.467590 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9e19b7-77b847565c-kr2r2"] Sep 14 10:11:02.479308 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.479288 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf"] Sep 14 10:11:02.482976 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:02.482957 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9e19b7-5ddcc88f46-snklf"] Sep 14 10:11:03.824907 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:03.824871 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3badb610-7f4b-4f50-a8d7-8c54f955ee1b" path="/var/lib/kubelet/pods/3badb610-7f4b-4f50-a8d7-8c54f955ee1b/volumes" Sep 14 10:11:03.825247 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:11:03.825231 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4" path="/var/lib/kubelet/pods/82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4/volumes" Sep 14 10:13:03.943117 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:13:03.943088 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:13:03.947916 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:13:03.947890 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:14:03.963844 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.963809 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g"] Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964466 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece13ac0-38b0-408e-a5a0-a4309aadd56f" containerName="maas-api" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964486 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece13ac0-38b0-408e-a5a0-a4309aadd56f" containerName="maas-api" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964504 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baba4bae-13e2-43e7-a5af-ad81a475d96b" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964513 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="baba4bae-13e2-43e7-a5af-ad81a475d96b" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964522 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="721657f5-f642-47b4-b14d-83edb98751fe" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964530 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="721657f5-f642-47b4-b14d-83edb98751fe" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964542 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964550 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964568 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5dba100-838a-49dc-bdee-9a0883e98396" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964577 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5dba100-838a-49dc-bdee-9a0883e98396" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964589 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf406175-8413-4320-9043-0188ffc52141" containerName="maas-api" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964597 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf406175-8413-4320-9043-0188ffc52141" containerName="maas-api" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964609 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ace602e-4cae-4e21-8b8c-efff41247ee7" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964617 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ace602e-4cae-4e21-8b8c-efff41247ee7" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964630 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f621568-0c5e-41ad-bb4b-f4841bdcbc26" containerName="maas-api" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964655 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f621568-0c5e-41ad-bb4b-f4841bdcbc26" containerName="maas-api" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964679 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf07770-24c1-4eaf-a696-de83cc22dcd2" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964687 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf07770-24c1-4eaf-a696-de83cc22dcd2" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964704 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6b37aa-2afd-49aa-aa1c-0d33d78abb88" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964713 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6b37aa-2afd-49aa-aa1c-0d33d78abb88" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964723 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc763f21-0c0e-4ae8-bc3b-ef835738f829" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964732 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc763f21-0c0e-4ae8-bc3b-ef835738f829" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964741 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a69e40-7e68-4e33-8ba7-ca49edde49cc" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964750 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a69e40-7e68-4e33-8ba7-ca49edde49cc" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964766 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3badb610-7f4b-4f50-a8d7-8c54f955ee1b" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964775 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3badb610-7f4b-4f50-a8d7-8c54f955ee1b" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964871 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a69e40-7e68-4e33-8ba7-ca49edde49cc" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964885 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ace602e-4cae-4e21-8b8c-efff41247ee7" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964896 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf406175-8413-4320-9043-0188ffc52141" containerName="maas-api" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964904 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc763f21-0c0e-4ae8-bc3b-ef835738f829" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964912 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf07770-24c1-4eaf-a696-de83cc22dcd2" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964920 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="baba4bae-13e2-43e7-a5af-ad81a475d96b" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964928 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f621568-0c5e-41ad-bb4b-f4841bdcbc26" containerName="maas-api" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964937 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="721657f5-f642-47b4-b14d-83edb98751fe" containerName="payload-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964944 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3badb610-7f4b-4f50-a8d7-8c54f955ee1b" containerName="payload-pre-processing" Sep 14 10:14:03.966544 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964955 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece13ac0-38b0-408e-a5a0-a4309aadd56f" containerName="maas-api" Sep 14 10:14:03.967758 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964965 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5dba100-838a-49dc-bdee-9a0883e98396" containerName="payload-pre-processing" Sep 14 10:14:03.967758 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964974 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6b37aa-2afd-49aa-aa1c-0d33d78abb88" containerName="payload-pre-processing" Sep 14 10:14:03.967758 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.964982 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f2b5fb-87d9-4472-9a7a-3ccaec51e3e4" containerName="payload-processing" Sep 14 10:14:03.968368 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.968348 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:03.971380 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.971358 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-78742b-serving-cert\"" Sep 14 10:14:03.977918 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:03.977882 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g"] Sep 14 10:14:04.045681 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.045649 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fpsg\" (UniqueName: \"kubernetes.io/projected/8a6270b1-4100-4ebd-b5e1-32a9655c9056-kube-api-access-6fpsg\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.045829 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.045692 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.045829 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.045778 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-metrics-certs\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.045829 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.045818 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6270b1-4100-4ebd-b5e1-32a9655c9056-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.146842 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.146808 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-metrics-certs\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.147007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.146862 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6270b1-4100-4ebd-b5e1-32a9655c9056-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.147007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.146928 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fpsg\" (UniqueName: \"kubernetes.io/projected/8a6270b1-4100-4ebd-b5e1-32a9655c9056-kube-api-access-6fpsg\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.147007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.146952 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.147135 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:14:04.147093 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-serving-cert: secret "maas-api-e2e-ipp-a-78742b-serving-cert" not found Sep 14 10:14:04.147183 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:14:04.147155 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-maas-api-tls podName:8a6270b1-4100-4ebd-b5e1-32a9655c9056 nodeName:}" failed. No retries permitted until 2026-09-14 10:14:04.647136854 +0000 UTC m=+1861.342262139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-maas-api-tls") pod "maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" (UID: "8a6270b1-4100-4ebd-b5e1-32a9655c9056") : secret "maas-api-e2e-ipp-a-78742b-serving-cert" not found Sep 14 10:14:04.147720 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.147696 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6270b1-4100-4ebd-b5e1-32a9655c9056-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.149370 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.149348 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-metrics-certs\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.160762 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.160739 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fpsg\" (UniqueName: \"kubernetes.io/projected/8a6270b1-4100-4ebd-b5e1-32a9655c9056-kube-api-access-6fpsg\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.639846 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.639816 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg"] Sep 14 10:14:04.643410 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.643392 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:14:04.645976 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.645933 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-78742b\"" Sep 14 10:14:04.646076 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.645991 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-78742b-dockercfg-tc4gg\"" Sep 14 10:14:04.651477 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.651348 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.653265 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.653239 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg"] Sep 14 10:14:04.654461 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.654430 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.752674 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.752619 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6bce831-9151-418f-9430-231ad16fb538-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg\" (UID: \"c6bce831-9151-418f-9430-231ad16fb538\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:14:04.752871 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.752711 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk5rc\" (UniqueName: \"kubernetes.io/projected/c6bce831-9151-418f-9430-231ad16fb538-kube-api-access-fk5rc\") pod \"payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg\" (UID: \"c6bce831-9151-418f-9430-231ad16fb538\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:14:04.853898 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.853861 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk5rc\" (UniqueName: \"kubernetes.io/projected/c6bce831-9151-418f-9430-231ad16fb538-kube-api-access-fk5rc\") pod \"payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg\" (UID: \"c6bce831-9151-418f-9430-231ad16fb538\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:14:04.854040 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.854021 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6bce831-9151-418f-9430-231ad16fb538-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg\" (UID: \"c6bce831-9151-418f-9430-231ad16fb538\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:14:04.854673 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.854631 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6bce831-9151-418f-9430-231ad16fb538-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg\" (UID: \"c6bce831-9151-418f-9430-231ad16fb538\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:14:04.862042 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.862021 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk5rc\" (UniqueName: \"kubernetes.io/projected/c6bce831-9151-418f-9430-231ad16fb538-kube-api-access-fk5rc\") pod \"payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg\" (UID: \"c6bce831-9151-418f-9430-231ad16fb538\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:14:04.881829 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.881808 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:04.957111 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:04.957078 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:14:05.153069 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:05.153044 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g"] Sep 14 10:14:05.154486 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:14:05.154463 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6270b1_4100_4ebd_b5e1_32a9655c9056.slice/crio-d0813cfd7df30f771150cff0aeb7cd283b415fcf235119bc043e64814fe3cdeb WatchSource:0}: Error finding container d0813cfd7df30f771150cff0aeb7cd283b415fcf235119bc043e64814fe3cdeb: Status 404 returned error can't find the container with id d0813cfd7df30f771150cff0aeb7cd283b415fcf235119bc043e64814fe3cdeb Sep 14 10:14:05.155860 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:05.155841 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 10:14:05.227374 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:05.227353 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg"] Sep 14 10:14:05.229519 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:14:05.229488 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bce831_9151_418f_9430_231ad16fb538.slice/crio-25ee863ee5da1eeddc197cdc603ec0168014f06fa4fb30ecb8aba2cc108a6e0e WatchSource:0}: Error finding container 25ee863ee5da1eeddc197cdc603ec0168014f06fa4fb30ecb8aba2cc108a6e0e: Status 404 returned error can't find the container with id 25ee863ee5da1eeddc197cdc603ec0168014f06fa4fb30ecb8aba2cc108a6e0e Sep 14 10:14:05.361319 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:05.361271 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" event={"ID":"c6bce831-9151-418f-9430-231ad16fb538","Type":"ContainerStarted","Data":"e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45"} Sep 14 10:14:05.361503 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:05.361342 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:14:05.361503 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:05.361369 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" event={"ID":"c6bce831-9151-418f-9430-231ad16fb538","Type":"ContainerStarted","Data":"25ee863ee5da1eeddc197cdc603ec0168014f06fa4fb30ecb8aba2cc108a6e0e"} Sep 14 10:14:05.362577 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:05.362548 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" event={"ID":"8a6270b1-4100-4ebd-b5e1-32a9655c9056","Type":"ContainerStarted","Data":"d0813cfd7df30f771150cff0aeb7cd283b415fcf235119bc043e64814fe3cdeb"} Sep 14 10:14:05.379250 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:05.379206 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" podStartSLOduration=1.379192251 podStartE2EDuration="1.379192251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:14:05.377932557 +0000 UTC m=+1862.073057854" watchObservedRunningTime="2026-09-14 10:14:05.379192251 +0000 UTC m=+1862.074317556" Sep 14 10:14:06.369493 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:06.369450 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" event={"ID":"8a6270b1-4100-4ebd-b5e1-32a9655c9056","Type":"ContainerStarted","Data":"bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162"} Sep 14 10:14:06.369909 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:06.369673 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:06.388154 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:06.388110 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" podStartSLOduration=2.931106825 podStartE2EDuration="3.388095871s" totalImagesPullingTime="456.989046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:14:06.387409395 +0000 UTC m=+1863.082534701" watchObservedRunningTime="2026-09-14 10:14:06.388095871 +0000 UTC m=+1863.083221175" Sep 14 10:14:12.379997 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:12.379969 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:14:16.371728 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:16.371694 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:14:34.738330 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:34.738296 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9"] Sep 14 10:14:34.742116 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:34.742098 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:14:34.744490 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:34.744472 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2c2a39\"" Sep 14 10:14:34.751688 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:34.751662 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9"] Sep 14 10:14:34.840596 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:34.840557 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qktf7\" (UniqueName: \"kubernetes.io/projected/ea72747d-540a-46b9-b97e-ee30264aa2a8-kube-api-access-qktf7\") pod \"payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9\" (UID: \"ea72747d-540a-46b9-b97e-ee30264aa2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:14:34.840790 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:34.840655 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea72747d-540a-46b9-b97e-ee30264aa2a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9\" (UID: \"ea72747d-540a-46b9-b97e-ee30264aa2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:14:34.941743 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:34.941708 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea72747d-540a-46b9-b97e-ee30264aa2a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9\" (UID: \"ea72747d-540a-46b9-b97e-ee30264aa2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:14:34.941921 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:34.941845 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qktf7\" (UniqueName: \"kubernetes.io/projected/ea72747d-540a-46b9-b97e-ee30264aa2a8-kube-api-access-qktf7\") pod \"payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9\" (UID: \"ea72747d-540a-46b9-b97e-ee30264aa2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:14:34.942420 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:34.942388 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea72747d-540a-46b9-b97e-ee30264aa2a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9\" (UID: \"ea72747d-540a-46b9-b97e-ee30264aa2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:14:34.950110 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:34.950083 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qktf7\" (UniqueName: \"kubernetes.io/projected/ea72747d-540a-46b9-b97e-ee30264aa2a8-kube-api-access-qktf7\") pod \"payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9\" (UID: \"ea72747d-540a-46b9-b97e-ee30264aa2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:14:35.053615 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:35.053533 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:14:35.322129 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:35.322082 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9"] Sep 14 10:14:35.324411 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:14:35.324374 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea72747d_540a_46b9_b97e_ee30264aa2a8.slice/crio-d780d9ea3a5e86dfd77a6434ea57cd3004c3cd894370b142e03f2696119d6edd WatchSource:0}: Error finding container d780d9ea3a5e86dfd77a6434ea57cd3004c3cd894370b142e03f2696119d6edd: Status 404 returned error can't find the container with id d780d9ea3a5e86dfd77a6434ea57cd3004c3cd894370b142e03f2696119d6edd Sep 14 10:14:35.517946 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:35.517907 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" event={"ID":"ea72747d-540a-46b9-b97e-ee30264aa2a8","Type":"ContainerStarted","Data":"4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb"} Sep 14 10:14:35.518127 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:35.517963 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" event={"ID":"ea72747d-540a-46b9-b97e-ee30264aa2a8","Type":"ContainerStarted","Data":"d780d9ea3a5e86dfd77a6434ea57cd3004c3cd894370b142e03f2696119d6edd"} Sep 14 10:14:35.518127 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:35.517991 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:14:35.536190 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:35.536138 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" podStartSLOduration=1.5361204979999998 podStartE2EDuration="1.536120498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:14:35.535510386 +0000 UTC m=+1892.230635696" watchObservedRunningTime="2026-09-14 10:14:35.536120498 +0000 UTC m=+1892.231245806" Sep 14 10:14:46.526800 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:14:46.526769 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:15:00.160451 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:00.160413 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54"] Sep 14 10:15:00.164247 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:00.164231 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54" Sep 14 10:15:00.167880 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:00.167854 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\"" Sep 14 10:15:00.173707 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:00.173688 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54" Sep 14 10:15:00.201350 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:00.201307 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54"] Sep 14 10:15:00.453497 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:00.453446 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54"] Sep 14 10:15:00.455717 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:15:00.455690 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d3f97b_637d_4800_8d4f_a355de7d942c.slice/crio-25402bc8bde4c33535303cd14977151049c93047f2d4b1178e84d132a5ec1d5f WatchSource:0}: Error finding container 25402bc8bde4c33535303cd14977151049c93047f2d4b1178e84d132a5ec1d5f: Status 404 returned error can't find the container with id 25402bc8bde4c33535303cd14977151049c93047f2d4b1178e84d132a5ec1d5f Sep 14 10:15:00.648834 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:00.648806 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54" event={"ID":"06d3f97b-637d-4800-8d4f-a355de7d942c","Type":"ContainerStarted","Data":"25402bc8bde4c33535303cd14977151049c93047f2d4b1178e84d132a5ec1d5f"} Sep 14 10:15:01.655682 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:01.655591 2550 generic.go:356] "Generic (PLEG): container finished" podID="06d3f97b-637d-4800-8d4f-a355de7d942c" containerID="ce9e6acc4822530b0092dd1902d1a6a5be846fa04a1206282b96d42503922ce1" exitCode=0 Sep 14 10:15:01.656041 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:01.655679 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54" event={"ID":"06d3f97b-637d-4800-8d4f-a355de7d942c","Type":"ContainerDied","Data":"ce9e6acc4822530b0092dd1902d1a6a5be846fa04a1206282b96d42503922ce1"} Sep 14 10:15:02.808495 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:02.808472 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54" Sep 14 10:15:03.668270 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:03.668239 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54" Sep 14 10:15:03.668440 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:03.668278 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54" event={"ID":"06d3f97b-637d-4800-8d4f-a355de7d942c","Type":"ContainerDied","Data":"25402bc8bde4c33535303cd14977151049c93047f2d4b1178e84d132a5ec1d5f"} Sep 14 10:15:03.668440 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:03.668329 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25402bc8bde4c33535303cd14977151049c93047f2d4b1178e84d132a5ec1d5f" Sep 14 10:15:32.083534 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:32.083503 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg_c6bce831-9151-418f-9430-231ad16fb538/payload-processing/0.log" Sep 14 10:15:39.577306 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.577268 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7"] Sep 14 10:15:39.578003 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.577985 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d3f97b-637d-4800-8d4f-a355de7d942c" containerName="cleanup" Sep 14 10:15:39.578092 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.578006 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d3f97b-637d-4800-8d4f-a355de7d942c" containerName="cleanup" Sep 14 10:15:39.578155 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.578106 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d3f97b-637d-4800-8d4f-a355de7d942c" containerName="cleanup" Sep 14 10:15:39.581723 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.581704 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.585162 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.585136 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-51f250-serving-cert\"" Sep 14 10:15:39.596903 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.596878 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7"] Sep 14 10:15:39.737470 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.737423 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkrcn\" (UniqueName: \"kubernetes.io/projected/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-kube-api-access-lkrcn\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.737695 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.737474 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.737695 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.737604 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.737695 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.737675 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.838341 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.838251 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.838526 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.838353 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.838526 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.838398 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.838526 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.838462 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkrcn\" (UniqueName: \"kubernetes.io/projected/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-kube-api-access-lkrcn\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.838526 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:15:39.838484 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-serving-cert: secret "maas-api-e2e-ipp-cleanup-51f250-serving-cert" not found Sep 14 10:15:39.838766 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:15:39.838561 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-maas-api-tls podName:eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b nodeName:}" failed. No retries permitted until 2026-09-14 10:15:40.338544928 +0000 UTC m=+1957.033670211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" (UID: "eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b") : secret "maas-api-e2e-ipp-cleanup-51f250-serving-cert" not found Sep 14 10:15:39.839072 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.839053 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.840968 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.840946 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:39.846690 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:39.846659 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkrcn\" (UniqueName: \"kubernetes.io/projected/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-kube-api-access-lkrcn\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:40.234921 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.234882 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb"] Sep 14 10:15:40.238690 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.238670 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:15:40.241366 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.241340 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-51f250\"" Sep 14 10:15:40.241692 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.241670 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-51f250-dockercfg-sczpm\"" Sep 14 10:15:40.250397 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.250373 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb"] Sep 14 10:15:40.344087 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.344044 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52ae09a-a490-4397-9a28-e0cadf6fe734-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb\" (UID: \"d52ae09a-a490-4397-9a28-e0cadf6fe734\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:15:40.344299 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.344128 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjn7\" (UniqueName: \"kubernetes.io/projected/d52ae09a-a490-4397-9a28-e0cadf6fe734-kube-api-access-4xjn7\") pod \"payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb\" (UID: \"d52ae09a-a490-4397-9a28-e0cadf6fe734\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:15:40.344299 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.344216 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:40.346679 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.346655 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:40.445241 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.445210 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52ae09a-a490-4397-9a28-e0cadf6fe734-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb\" (UID: \"d52ae09a-a490-4397-9a28-e0cadf6fe734\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:15:40.445399 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.445261 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjn7\" (UniqueName: \"kubernetes.io/projected/d52ae09a-a490-4397-9a28-e0cadf6fe734-kube-api-access-4xjn7\") pod \"payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb\" (UID: \"d52ae09a-a490-4397-9a28-e0cadf6fe734\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:15:40.445944 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.445924 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52ae09a-a490-4397-9a28-e0cadf6fe734-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb\" (UID: \"d52ae09a-a490-4397-9a28-e0cadf6fe734\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:15:40.455533 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.455513 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjn7\" (UniqueName: \"kubernetes.io/projected/d52ae09a-a490-4397-9a28-e0cadf6fe734-kube-api-access-4xjn7\") pod \"payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb\" (UID: \"d52ae09a-a490-4397-9a28-e0cadf6fe734\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:15:40.503533 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.503470 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:40.550624 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.550588 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:15:40.794708 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.794681 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7"] Sep 14 10:15:40.796081 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:15:40.796053 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae23d1f_b98e_4ea9_a2ab_2b30cff4bc8b.slice/crio-1b552c6f6348efb09f9884cec38f3dbe89fca929f15aa386c0b5b48cee2f6b4c WatchSource:0}: Error finding container 1b552c6f6348efb09f9884cec38f3dbe89fca929f15aa386c0b5b48cee2f6b4c: Status 404 returned error can't find the container with id 1b552c6f6348efb09f9884cec38f3dbe89fca929f15aa386c0b5b48cee2f6b4c Sep 14 10:15:40.835966 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.835942 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb"] Sep 14 10:15:40.837166 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:15:40.837145 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52ae09a_a490_4397_9a28_e0cadf6fe734.slice/crio-369e21dedb6f0af74727c91094b16dd1729d72a0706cff7dc1c4f913592c3c7a WatchSource:0}: Error finding container 369e21dedb6f0af74727c91094b16dd1729d72a0706cff7dc1c4f913592c3c7a: Status 404 returned error can't find the container with id 369e21dedb6f0af74727c91094b16dd1729d72a0706cff7dc1c4f913592c3c7a Sep 14 10:15:40.870680 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.870654 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" event={"ID":"d52ae09a-a490-4397-9a28-e0cadf6fe734","Type":"ContainerStarted","Data":"369e21dedb6f0af74727c91094b16dd1729d72a0706cff7dc1c4f913592c3c7a"} Sep 14 10:15:40.871857 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:40.871834 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" event={"ID":"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b","Type":"ContainerStarted","Data":"1b552c6f6348efb09f9884cec38f3dbe89fca929f15aa386c0b5b48cee2f6b4c"} Sep 14 10:15:41.877962 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:41.877923 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" event={"ID":"d52ae09a-a490-4397-9a28-e0cadf6fe734","Type":"ContainerStarted","Data":"35487d6230aea0fe333892faa5ce3cd83cd38c030ef801d3dc1b33b69c6d6f24"} Sep 14 10:15:41.878381 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:41.878015 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:15:41.879512 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:41.879486 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" event={"ID":"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b","Type":"ContainerStarted","Data":"89d8fe9202ee175bf2294d327fc2305d69eba4bce465d481e9312dc162f0ab8b"} Sep 14 10:15:41.879670 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:41.879527 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:41.895454 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:41.895412 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" podStartSLOduration=1.895399026 podStartE2EDuration="1.895399026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:15:41.894848454 +0000 UTC m=+1958.589973762" watchObservedRunningTime="2026-09-14 10:15:41.895399026 +0000 UTC m=+1958.590524330" Sep 14 10:15:41.913424 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:41.913381 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" podStartSLOduration=2.507608796 podStartE2EDuration="2.913368793s" totalImagesPullingTime="405.759997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:15:41.911362874 +0000 UTC m=+1958.606488176" watchObservedRunningTime="2026-09-14 10:15:41.913368793 +0000 UTC m=+1958.608494098" Sep 14 10:15:47.889965 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:47.889938 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:15:52.887063 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:15:52.887035 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:16:17.843533 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:17.843444 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7"] Sep 14 10:16:17.844179 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:17.843645 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" podUID="eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" containerName="maas-api" containerID="cri-o://89d8fe9202ee175bf2294d327fc2305d69eba4bce465d481e9312dc162f0ab8b" gracePeriod=30 Sep 14 10:16:17.885040 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:17.885006 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" podUID="eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.73:8443/health\": dial tcp 10.132.0.73:8443: connect: connection refused" Sep 14 10:16:17.909798 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:17.909768 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb"] Sep 14 10:16:17.910007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:17.909978 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" podUID="d52ae09a-a490-4397-9a28-e0cadf6fe734" containerName="payload-processing" containerID="cri-o://35487d6230aea0fe333892faa5ce3cd83cd38c030ef801d3dc1b33b69c6d6f24" gracePeriod=30 Sep 14 10:16:18.073537 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.073236 2550 generic.go:356] "Generic (PLEG): container finished" podID="d52ae09a-a490-4397-9a28-e0cadf6fe734" containerID="35487d6230aea0fe333892faa5ce3cd83cd38c030ef801d3dc1b33b69c6d6f24" exitCode=0 Sep 14 10:16:18.073537 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.073290 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" event={"ID":"d52ae09a-a490-4397-9a28-e0cadf6fe734","Type":"ContainerDied","Data":"35487d6230aea0fe333892faa5ce3cd83cd38c030ef801d3dc1b33b69c6d6f24"} Sep 14 10:16:18.075978 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.075721 2550 generic.go:356] "Generic (PLEG): container finished" podID="eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" containerID="89d8fe9202ee175bf2294d327fc2305d69eba4bce465d481e9312dc162f0ab8b" exitCode=0 Sep 14 10:16:18.075978 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.075817 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" event={"ID":"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b","Type":"ContainerDied","Data":"89d8fe9202ee175bf2294d327fc2305d69eba4bce465d481e9312dc162f0ab8b"} Sep 14 10:16:18.171536 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.171512 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:16:18.197593 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.197568 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:16:18.303258 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.303226 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xjn7\" (UniqueName: \"kubernetes.io/projected/d52ae09a-a490-4397-9a28-e0cadf6fe734-kube-api-access-4xjn7\") pod \"d52ae09a-a490-4397-9a28-e0cadf6fe734\" (UID: \"d52ae09a-a490-4397-9a28-e0cadf6fe734\") " Sep 14 10:16:18.303258 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.303260 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkrcn\" (UniqueName: \"kubernetes.io/projected/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-kube-api-access-lkrcn\") pod \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " Sep 14 10:16:18.303495 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.303330 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-maas-api-tls\") pod \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " Sep 14 10:16:18.303495 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.303381 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52ae09a-a490-4397-9a28-e0cadf6fe734-plugins-config-volume\") pod \"d52ae09a-a490-4397-9a28-e0cadf6fe734\" (UID: \"d52ae09a-a490-4397-9a28-e0cadf6fe734\") " Sep 14 10:16:18.303495 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.303423 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-metrics-certs\") pod \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " Sep 14 10:16:18.303495 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.303451 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-openshift-service-ca\") pod \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\" (UID: \"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b\") " Sep 14 10:16:18.303826 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.303793 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52ae09a-a490-4397-9a28-e0cadf6fe734-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d52ae09a-a490-4397-9a28-e0cadf6fe734" (UID: "d52ae09a-a490-4397-9a28-e0cadf6fe734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:16:18.303975 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.303932 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" (UID: "eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:16:18.305452 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.305432 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" (UID: "eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:16:18.305524 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.305503 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-kube-api-access-lkrcn" (OuterVolumeSpecName: "kube-api-access-lkrcn") pod "eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" (UID: "eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b"). InnerVolumeSpecName "kube-api-access-lkrcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:16:18.305815 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.305787 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" (UID: "eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:16:18.305923 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.305797 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52ae09a-a490-4397-9a28-e0cadf6fe734-kube-api-access-4xjn7" (OuterVolumeSpecName: "kube-api-access-4xjn7") pod "d52ae09a-a490-4397-9a28-e0cadf6fe734" (UID: "d52ae09a-a490-4397-9a28-e0cadf6fe734"). InnerVolumeSpecName "kube-api-access-4xjn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:16:18.404664 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.404573 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xjn7\" (UniqueName: \"kubernetes.io/projected/d52ae09a-a490-4397-9a28-e0cadf6fe734-kube-api-access-4xjn7\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:18.404664 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.404596 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkrcn\" (UniqueName: \"kubernetes.io/projected/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-kube-api-access-lkrcn\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:18.404664 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.404606 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:18.404664 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.404615 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52ae09a-a490-4397-9a28-e0cadf6fe734-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:18.404664 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.404624 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:18.404664 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:18.404656 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:19.082316 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.082286 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" Sep 14 10:16:19.082789 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.082291 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb" event={"ID":"d52ae09a-a490-4397-9a28-e0cadf6fe734","Type":"ContainerDied","Data":"369e21dedb6f0af74727c91094b16dd1729d72a0706cff7dc1c4f913592c3c7a"} Sep 14 10:16:19.082789 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.082422 2550 scope.go:117] "RemoveContainer" containerID="35487d6230aea0fe333892faa5ce3cd83cd38c030ef801d3dc1b33b69c6d6f24" Sep 14 10:16:19.083832 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.083769 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" Sep 14 10:16:19.083908 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.083764 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7" event={"ID":"eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b","Type":"ContainerDied","Data":"1b552c6f6348efb09f9884cec38f3dbe89fca929f15aa386c0b5b48cee2f6b4c"} Sep 14 10:16:19.092099 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.092082 2550 scope.go:117] "RemoveContainer" containerID="89d8fe9202ee175bf2294d327fc2305d69eba4bce465d481e9312dc162f0ab8b" Sep 14 10:16:19.128959 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.128900 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7"] Sep 14 10:16:19.138018 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.137998 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-51f250-7b657c5897-zbsv7"] Sep 14 10:16:19.152722 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.152697 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb"] Sep 14 10:16:19.154445 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.154425 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-51f250-67868ff8b5-gx6nb"] Sep 14 10:16:19.831358 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.831316 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52ae09a-a490-4397-9a28-e0cadf6fe734" path="/var/lib/kubelet/pods/d52ae09a-a490-4397-9a28-e0cadf6fe734/volumes" Sep 14 10:16:19.832019 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:19.831992 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" path="/var/lib/kubelet/pods/eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b/volumes" Sep 14 10:16:45.450642 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.450598 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g"] Sep 14 10:16:45.451075 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.450832 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" podUID="8a6270b1-4100-4ebd-b5e1-32a9655c9056" containerName="maas-api" containerID="cri-o://bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162" gracePeriod=30 Sep 14 10:16:45.565700 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.565667 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg"] Sep 14 10:16:45.566095 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.565887 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" podUID="c6bce831-9151-418f-9430-231ad16fb538" containerName="payload-processing" containerID="cri-o://e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45" gracePeriod=30 Sep 14 10:16:45.710984 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.710922 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:16:45.768178 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.768152 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-maas-api-tls\") pod \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " Sep 14 10:16:45.768351 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.768200 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6270b1-4100-4ebd-b5e1-32a9655c9056-openshift-service-ca\") pod \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " Sep 14 10:16:45.768351 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.768259 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-metrics-certs\") pod \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " Sep 14 10:16:45.768351 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.768304 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fpsg\" (UniqueName: \"kubernetes.io/projected/8a6270b1-4100-4ebd-b5e1-32a9655c9056-kube-api-access-6fpsg\") pod \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\" (UID: \"8a6270b1-4100-4ebd-b5e1-32a9655c9056\") " Sep 14 10:16:45.768703 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.768667 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6270b1-4100-4ebd-b5e1-32a9655c9056-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a6270b1-4100-4ebd-b5e1-32a9655c9056" (UID: "8a6270b1-4100-4ebd-b5e1-32a9655c9056"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:16:45.770683 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.770627 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6270b1-4100-4ebd-b5e1-32a9655c9056-kube-api-access-6fpsg" (OuterVolumeSpecName: "kube-api-access-6fpsg") pod "8a6270b1-4100-4ebd-b5e1-32a9655c9056" (UID: "8a6270b1-4100-4ebd-b5e1-32a9655c9056"). InnerVolumeSpecName "kube-api-access-6fpsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:16:45.776926 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.770934 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a6270b1-4100-4ebd-b5e1-32a9655c9056" (UID: "8a6270b1-4100-4ebd-b5e1-32a9655c9056"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:16:45.776926 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.770947 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a6270b1-4100-4ebd-b5e1-32a9655c9056" (UID: "8a6270b1-4100-4ebd-b5e1-32a9655c9056"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:16:45.820773 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.820746 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:16:45.869079 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.869054 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk5rc\" (UniqueName: \"kubernetes.io/projected/c6bce831-9151-418f-9430-231ad16fb538-kube-api-access-fk5rc\") pod \"c6bce831-9151-418f-9430-231ad16fb538\" (UID: \"c6bce831-9151-418f-9430-231ad16fb538\") " Sep 14 10:16:45.869221 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.869142 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6bce831-9151-418f-9430-231ad16fb538-plugins-config-volume\") pod \"c6bce831-9151-418f-9430-231ad16fb538\" (UID: \"c6bce831-9151-418f-9430-231ad16fb538\") " Sep 14 10:16:45.869362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.869337 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:45.869362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.869354 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6270b1-4100-4ebd-b5e1-32a9655c9056-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:45.869505 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.869367 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6270b1-4100-4ebd-b5e1-32a9655c9056-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:45.869505 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.869380 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fpsg\" (UniqueName: \"kubernetes.io/projected/8a6270b1-4100-4ebd-b5e1-32a9655c9056-kube-api-access-6fpsg\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:45.869505 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.869469 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6bce831-9151-418f-9430-231ad16fb538-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6bce831-9151-418f-9430-231ad16fb538" (UID: "c6bce831-9151-418f-9430-231ad16fb538"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:16:45.871105 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.871085 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6bce831-9151-418f-9430-231ad16fb538-kube-api-access-fk5rc" (OuterVolumeSpecName: "kube-api-access-fk5rc") pod "c6bce831-9151-418f-9430-231ad16fb538" (UID: "c6bce831-9151-418f-9430-231ad16fb538"). InnerVolumeSpecName "kube-api-access-fk5rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:16:45.970442 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.970361 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6bce831-9151-418f-9430-231ad16fb538-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:45.970442 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:45.970389 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk5rc\" (UniqueName: \"kubernetes.io/projected/c6bce831-9151-418f-9430-231ad16fb538-kube-api-access-fk5rc\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:16:46.228052 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.227962 2550 generic.go:356] "Generic (PLEG): container finished" podID="8a6270b1-4100-4ebd-b5e1-32a9655c9056" containerID="bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162" exitCode=0 Sep 14 10:16:46.228052 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.228007 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" Sep 14 10:16:46.228052 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.228040 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" event={"ID":"8a6270b1-4100-4ebd-b5e1-32a9655c9056","Type":"ContainerDied","Data":"bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162"} Sep 14 10:16:46.228320 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.228097 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g" event={"ID":"8a6270b1-4100-4ebd-b5e1-32a9655c9056","Type":"ContainerDied","Data":"d0813cfd7df30f771150cff0aeb7cd283b415fcf235119bc043e64814fe3cdeb"} Sep 14 10:16:46.228320 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.228120 2550 scope.go:117] "RemoveContainer" containerID="bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162" Sep 14 10:16:46.229591 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.229568 2550 generic.go:356] "Generic (PLEG): container finished" podID="c6bce831-9151-418f-9430-231ad16fb538" containerID="e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45" exitCode=0 Sep 14 10:16:46.229795 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.229606 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" Sep 14 10:16:46.229795 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.229608 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" event={"ID":"c6bce831-9151-418f-9430-231ad16fb538","Type":"ContainerDied","Data":"e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45"} Sep 14 10:16:46.229795 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.229678 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg" event={"ID":"c6bce831-9151-418f-9430-231ad16fb538","Type":"ContainerDied","Data":"25ee863ee5da1eeddc197cdc603ec0168014f06fa4fb30ecb8aba2cc108a6e0e"} Sep 14 10:16:46.237870 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.237857 2550 scope.go:117] "RemoveContainer" containerID="bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162" Sep 14 10:16:46.238164 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:16:46.238145 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162\": container with ID starting with bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162 not found: ID does not exist" containerID="bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162" Sep 14 10:16:46.238236 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.238179 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162"} err="failed to get container status \"bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162\": rpc error: code = NotFound desc = could not find container \"bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162\": container with ID starting with bd1799ac996436415ed65dbd302073377809c04eeb4d8fdf39a9081d4242a162 not found: ID does not exist" Sep 14 10:16:46.238236 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.238199 2550 scope.go:117] "RemoveContainer" containerID="e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45" Sep 14 10:16:46.245908 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.245890 2550 scope.go:117] "RemoveContainer" containerID="e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45" Sep 14 10:16:46.246169 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:16:46.246147 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45\": container with ID starting with e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45 not found: ID does not exist" containerID="e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45" Sep 14 10:16:46.246226 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.246175 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45"} err="failed to get container status \"e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45\": rpc error: code = NotFound desc = could not find container \"e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45\": container with ID starting with e5be51d90ebe0bd659e76d9a8759292d494f91997da1d856ad4f94b25f6b4c45 not found: ID does not exist" Sep 14 10:16:46.251662 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.251625 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g"] Sep 14 10:16:46.259492 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.259472 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78742b-7bb8cb4d87-flt6g"] Sep 14 10:16:46.280177 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.280153 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg"] Sep 14 10:16:46.294436 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:46.294409 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78742b-865466cd5c-btcmg"] Sep 14 10:16:47.825017 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:47.824983 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6270b1-4100-4ebd-b5e1-32a9655c9056" path="/var/lib/kubelet/pods/8a6270b1-4100-4ebd-b5e1-32a9655c9056/volumes" Sep 14 10:16:47.825536 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:16:47.825516 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6bce831-9151-418f-9430-231ad16fb538" path="/var/lib/kubelet/pods/c6bce831-9151-418f-9430-231ad16fb538/volumes" Sep 14 10:17:45.850465 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:45.850419 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9"] Sep 14 10:17:45.851032 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:45.850977 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" podUID="ea72747d-540a-46b9-b97e-ee30264aa2a8" containerName="payload-pre-processing" containerID="cri-o://4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb" gracePeriod=30 Sep 14 10:17:46.123936 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.123910 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:17:46.215027 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.214989 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea72747d-540a-46b9-b97e-ee30264aa2a8-plugins-config-volume\") pod \"ea72747d-540a-46b9-b97e-ee30264aa2a8\" (UID: \"ea72747d-540a-46b9-b97e-ee30264aa2a8\") " Sep 14 10:17:46.215322 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.215044 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qktf7\" (UniqueName: \"kubernetes.io/projected/ea72747d-540a-46b9-b97e-ee30264aa2a8-kube-api-access-qktf7\") pod \"ea72747d-540a-46b9-b97e-ee30264aa2a8\" (UID: \"ea72747d-540a-46b9-b97e-ee30264aa2a8\") " Sep 14 10:17:46.215406 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.215341 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea72747d-540a-46b9-b97e-ee30264aa2a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea72747d-540a-46b9-b97e-ee30264aa2a8" (UID: "ea72747d-540a-46b9-b97e-ee30264aa2a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:17:46.217272 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.217243 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea72747d-540a-46b9-b97e-ee30264aa2a8-kube-api-access-qktf7" (OuterVolumeSpecName: "kube-api-access-qktf7") pod "ea72747d-540a-46b9-b97e-ee30264aa2a8" (UID: "ea72747d-540a-46b9-b97e-ee30264aa2a8"). InnerVolumeSpecName "kube-api-access-qktf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:17:46.316697 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.316670 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea72747d-540a-46b9-b97e-ee30264aa2a8-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:17:46.316697 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.316694 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qktf7\" (UniqueName: \"kubernetes.io/projected/ea72747d-540a-46b9-b97e-ee30264aa2a8-kube-api-access-qktf7\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:17:46.532190 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.532153 2550 generic.go:356] "Generic (PLEG): container finished" podID="ea72747d-540a-46b9-b97e-ee30264aa2a8" containerID="4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb" exitCode=0 Sep 14 10:17:46.532373 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.532198 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" Sep 14 10:17:46.532373 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.532238 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" event={"ID":"ea72747d-540a-46b9-b97e-ee30264aa2a8","Type":"ContainerDied","Data":"4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb"} Sep 14 10:17:46.532373 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.532288 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9" event={"ID":"ea72747d-540a-46b9-b97e-ee30264aa2a8","Type":"ContainerDied","Data":"d780d9ea3a5e86dfd77a6434ea57cd3004c3cd894370b142e03f2696119d6edd"} Sep 14 10:17:46.532373 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.532320 2550 scope.go:117] "RemoveContainer" containerID="4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb" Sep 14 10:17:46.542515 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.542499 2550 scope.go:117] "RemoveContainer" containerID="4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb" Sep 14 10:17:46.542827 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:17:46.542806 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb\": container with ID starting with 4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb not found: ID does not exist" containerID="4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb" Sep 14 10:17:46.542932 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.542835 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb"} err="failed to get container status \"4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb\": rpc error: code = NotFound desc = could not find container \"4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb\": container with ID starting with 4979163b34f4cd5ac4fcf959667dec1ef24e9680d2e4906f278e174e23dac5eb not found: ID does not exist" Sep 14 10:17:46.555952 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.555926 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9"] Sep 14 10:17:46.557613 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:46.557594 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c2a39-9979fbdf4-96ps9"] Sep 14 10:17:47.824383 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:17:47.824352 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea72747d-540a-46b9-b97e-ee30264aa2a8" path="/var/lib/kubelet/pods/ea72747d-540a-46b9-b97e-ee30264aa2a8/volumes" Sep 14 10:18:03.776253 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.776207 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj"] Sep 14 10:18:03.777152 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.777099 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" podUID="563cac49-ae20-4251-8a35-1e7f0aea1846" containerName="maas-api" containerID="cri-o://5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549" gracePeriod=30 Sep 14 10:18:03.806342 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.806318 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54"] Sep 14 10:18:03.808947 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.808926 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-51aefd-29823015-4mz54"] Sep 14 10:18:03.825366 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.825338 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d3f97b-637d-4800-8d4f-a355de7d942c" path="/var/lib/kubelet/pods/06d3f97b-637d-4800-8d4f-a355de7d942c/volumes" Sep 14 10:18:03.844152 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.844127 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf"] Sep 14 10:18:03.844346 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.844321 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" podUID="19989a8b-8523-4b16-90a2-db5a9b372f2e" containerName="payload-processing" containerID="cri-o://60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a" gracePeriod=30 Sep 14 10:18:03.850191 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.850164 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9"] Sep 14 10:18:03.850400 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.850355 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" podUID="6c3d860d-6ef8-45f8-9cae-789f39846cdf" containerName="payload-pre-processing" containerID="cri-o://74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0" gracePeriod=30 Sep 14 10:18:03.992994 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.992867 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:18:03.998753 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:03.998729 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:18:04.126805 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.126781 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:18:04.145144 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.145123 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:18:04.147962 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.147944 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:18:04.181156 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.181132 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-metrics-certs\") pod \"563cac49-ae20-4251-8a35-1e7f0aea1846\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " Sep 14 10:18:04.181314 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.181192 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zbmw\" (UniqueName: \"kubernetes.io/projected/19989a8b-8523-4b16-90a2-db5a9b372f2e-kube-api-access-6zbmw\") pod \"19989a8b-8523-4b16-90a2-db5a9b372f2e\" (UID: \"19989a8b-8523-4b16-90a2-db5a9b372f2e\") " Sep 14 10:18:04.181314 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.181248 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlltl\" (UniqueName: \"kubernetes.io/projected/563cac49-ae20-4251-8a35-1e7f0aea1846-kube-api-access-dlltl\") pod \"563cac49-ae20-4251-8a35-1e7f0aea1846\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " Sep 14 10:18:04.181314 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.181307 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19989a8b-8523-4b16-90a2-db5a9b372f2e-plugins-config-volume\") pod \"19989a8b-8523-4b16-90a2-db5a9b372f2e\" (UID: \"19989a8b-8523-4b16-90a2-db5a9b372f2e\") " Sep 14 10:18:04.181493 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.181354 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3d860d-6ef8-45f8-9cae-789f39846cdf-plugins-config-volume\") pod \"6c3d860d-6ef8-45f8-9cae-789f39846cdf\" (UID: \"6c3d860d-6ef8-45f8-9cae-789f39846cdf\") " Sep 14 10:18:04.181493 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.181387 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-maas-api-tls\") pod \"563cac49-ae20-4251-8a35-1e7f0aea1846\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " Sep 14 10:18:04.181493 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.181411 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563cac49-ae20-4251-8a35-1e7f0aea1846-openshift-service-ca\") pod \"563cac49-ae20-4251-8a35-1e7f0aea1846\" (UID: \"563cac49-ae20-4251-8a35-1e7f0aea1846\") " Sep 14 10:18:04.181493 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.181450 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqzkd\" (UniqueName: \"kubernetes.io/projected/6c3d860d-6ef8-45f8-9cae-789f39846cdf-kube-api-access-pqzkd\") pod \"6c3d860d-6ef8-45f8-9cae-789f39846cdf\" (UID: \"6c3d860d-6ef8-45f8-9cae-789f39846cdf\") " Sep 14 10:18:04.181789 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.181759 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19989a8b-8523-4b16-90a2-db5a9b372f2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19989a8b-8523-4b16-90a2-db5a9b372f2e" (UID: "19989a8b-8523-4b16-90a2-db5a9b372f2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:18:04.182166 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.182034 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3d860d-6ef8-45f8-9cae-789f39846cdf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c3d860d-6ef8-45f8-9cae-789f39846cdf" (UID: "6c3d860d-6ef8-45f8-9cae-789f39846cdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:18:04.182284 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.182196 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563cac49-ae20-4251-8a35-1e7f0aea1846-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "563cac49-ae20-4251-8a35-1e7f0aea1846" (UID: "563cac49-ae20-4251-8a35-1e7f0aea1846"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:18:04.183681 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.183614 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3d860d-6ef8-45f8-9cae-789f39846cdf-kube-api-access-pqzkd" (OuterVolumeSpecName: "kube-api-access-pqzkd") pod "6c3d860d-6ef8-45f8-9cae-789f39846cdf" (UID: "6c3d860d-6ef8-45f8-9cae-789f39846cdf"). InnerVolumeSpecName "kube-api-access-pqzkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:18:04.183863 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.183839 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "563cac49-ae20-4251-8a35-1e7f0aea1846" (UID: "563cac49-ae20-4251-8a35-1e7f0aea1846"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:18:04.183943 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.183855 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19989a8b-8523-4b16-90a2-db5a9b372f2e-kube-api-access-6zbmw" (OuterVolumeSpecName: "kube-api-access-6zbmw") pod "19989a8b-8523-4b16-90a2-db5a9b372f2e" (UID: "19989a8b-8523-4b16-90a2-db5a9b372f2e"). InnerVolumeSpecName "kube-api-access-6zbmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:18:04.184027 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.184004 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563cac49-ae20-4251-8a35-1e7f0aea1846-kube-api-access-dlltl" (OuterVolumeSpecName: "kube-api-access-dlltl") pod "563cac49-ae20-4251-8a35-1e7f0aea1846" (UID: "563cac49-ae20-4251-8a35-1e7f0aea1846"). InnerVolumeSpecName "kube-api-access-dlltl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:18:04.184282 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.184243 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "563cac49-ae20-4251-8a35-1e7f0aea1846" (UID: "563cac49-ae20-4251-8a35-1e7f0aea1846"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:18:04.282291 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.282260 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19989a8b-8523-4b16-90a2-db5a9b372f2e-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:04.282291 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.282286 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3d860d-6ef8-45f8-9cae-789f39846cdf-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:04.282484 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.282301 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:04.282484 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.282314 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563cac49-ae20-4251-8a35-1e7f0aea1846-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:04.282484 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.282322 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqzkd\" (UniqueName: \"kubernetes.io/projected/6c3d860d-6ef8-45f8-9cae-789f39846cdf-kube-api-access-pqzkd\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:04.282484 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.282331 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563cac49-ae20-4251-8a35-1e7f0aea1846-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:04.282484 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.282339 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zbmw\" (UniqueName: \"kubernetes.io/projected/19989a8b-8523-4b16-90a2-db5a9b372f2e-kube-api-access-6zbmw\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:04.282484 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.282347 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlltl\" (UniqueName: \"kubernetes.io/projected/563cac49-ae20-4251-8a35-1e7f0aea1846-kube-api-access-dlltl\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:04.631088 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.631054 2550 generic.go:356] "Generic (PLEG): container finished" podID="6c3d860d-6ef8-45f8-9cae-789f39846cdf" containerID="74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0" exitCode=0 Sep 14 10:18:04.631278 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.631100 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" Sep 14 10:18:04.631278 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.631138 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" event={"ID":"6c3d860d-6ef8-45f8-9cae-789f39846cdf","Type":"ContainerDied","Data":"74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0"} Sep 14 10:18:04.631278 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.631196 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9" event={"ID":"6c3d860d-6ef8-45f8-9cae-789f39846cdf","Type":"ContainerDied","Data":"62a8bb66409014fca807812497719ddd8acc61fd9539aed5438d5dffca304cb2"} Sep 14 10:18:04.631278 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.631225 2550 scope.go:117] "RemoveContainer" containerID="74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0" Sep 14 10:18:04.632529 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.632469 2550 generic.go:356] "Generic (PLEG): container finished" podID="19989a8b-8523-4b16-90a2-db5a9b372f2e" containerID="60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a" exitCode=0 Sep 14 10:18:04.632529 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.632510 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" event={"ID":"19989a8b-8523-4b16-90a2-db5a9b372f2e","Type":"ContainerDied","Data":"60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a"} Sep 14 10:18:04.632529 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.632525 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" Sep 14 10:18:04.632825 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.632563 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf" event={"ID":"19989a8b-8523-4b16-90a2-db5a9b372f2e","Type":"ContainerDied","Data":"3cf81d8dcdacc40fa186a9f2dc03b0974ff1c6cf5cdda3731a81d94f7c85192c"} Sep 14 10:18:04.634296 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.634271 2550 generic.go:356] "Generic (PLEG): container finished" podID="563cac49-ae20-4251-8a35-1e7f0aea1846" containerID="5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549" exitCode=0 Sep 14 10:18:04.634418 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.634304 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" Sep 14 10:18:04.634418 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.634338 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" event={"ID":"563cac49-ae20-4251-8a35-1e7f0aea1846","Type":"ContainerDied","Data":"5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549"} Sep 14 10:18:04.634418 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.634378 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj" event={"ID":"563cac49-ae20-4251-8a35-1e7f0aea1846","Type":"ContainerDied","Data":"de39135b2289115b0e82fb830ad935425e41a487134f872c6a73befd7b0caa89"} Sep 14 10:18:04.641521 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.641504 2550 scope.go:117] "RemoveContainer" containerID="74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0" Sep 14 10:18:04.642032 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:18:04.641948 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0\": container with ID starting with 74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0 not found: ID does not exist" containerID="74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0" Sep 14 10:18:04.642032 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.641988 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0"} err="failed to get container status \"74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0\": rpc error: code = NotFound desc = could not find container \"74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0\": container with ID starting with 74754fdccdbfbe76fc55a195ef10dfd71d90eaa0b83487b12d2af1077d58dce0 not found: ID does not exist" Sep 14 10:18:04.642032 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.642012 2550 scope.go:117] "RemoveContainer" containerID="60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a" Sep 14 10:18:04.650379 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.650364 2550 scope.go:117] "RemoveContainer" containerID="60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a" Sep 14 10:18:04.650630 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:18:04.650614 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a\": container with ID starting with 60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a not found: ID does not exist" containerID="60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a" Sep 14 10:18:04.650690 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.650652 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a"} err="failed to get container status \"60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a\": rpc error: code = NotFound desc = could not find container \"60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a\": container with ID starting with 60b5baba2998bc987ab3f755ae67b1233c075270f5924769e69844f8b12bd05a not found: ID does not exist" Sep 14 10:18:04.650690 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.650671 2550 scope.go:117] "RemoveContainer" containerID="5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549" Sep 14 10:18:04.658426 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.658408 2550 scope.go:117] "RemoveContainer" containerID="5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549" Sep 14 10:18:04.658721 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:18:04.658703 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549\": container with ID starting with 5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549 not found: ID does not exist" containerID="5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549" Sep 14 10:18:04.658795 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.658730 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549"} err="failed to get container status \"5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549\": rpc error: code = NotFound desc = could not find container \"5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549\": container with ID starting with 5f83957b9de48f8dfdbe2ff82e58ee4f13fba5f12485a5e176930b4f83b38549 not found: ID does not exist" Sep 14 10:18:04.662846 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.662827 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9"] Sep 14 10:18:04.665832 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.665814 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-51aefd-5d78575c58-v5nq9"] Sep 14 10:18:04.680818 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.680797 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj"] Sep 14 10:18:04.688723 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.688698 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-51aefd-fcd555588-qqsjj"] Sep 14 10:18:04.701155 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.701134 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf"] Sep 14 10:18:04.707398 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:04.707379 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-51aefd-57b64ccfd7-hg4mf"] Sep 14 10:18:05.824139 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:05.824106 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19989a8b-8523-4b16-90a2-db5a9b372f2e" path="/var/lib/kubelet/pods/19989a8b-8523-4b16-90a2-db5a9b372f2e/volumes" Sep 14 10:18:05.824547 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:05.824489 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563cac49-ae20-4251-8a35-1e7f0aea1846" path="/var/lib/kubelet/pods/563cac49-ae20-4251-8a35-1e7f0aea1846/volumes" Sep 14 10:18:05.824835 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:05.824820 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c3d860d-6ef8-45f8-9cae-789f39846cdf" path="/var/lib/kubelet/pods/6c3d860d-6ef8-45f8-9cae-789f39846cdf/volumes" Sep 14 10:18:11.106566 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106531 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm"] Sep 14 10:18:11.106993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106917 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6270b1-4100-4ebd-b5e1-32a9655c9056" containerName="maas-api" Sep 14 10:18:11.106993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106929 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6270b1-4100-4ebd-b5e1-32a9655c9056" containerName="maas-api" Sep 14 10:18:11.106993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106940 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19989a8b-8523-4b16-90a2-db5a9b372f2e" containerName="payload-processing" Sep 14 10:18:11.106993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106946 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="19989a8b-8523-4b16-90a2-db5a9b372f2e" containerName="payload-processing" Sep 14 10:18:11.106993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106955 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563cac49-ae20-4251-8a35-1e7f0aea1846" containerName="maas-api" Sep 14 10:18:11.106993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106961 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="563cac49-ae20-4251-8a35-1e7f0aea1846" containerName="maas-api" Sep 14 10:18:11.106993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106972 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6bce831-9151-418f-9430-231ad16fb538" containerName="payload-processing" Sep 14 10:18:11.106993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106977 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6bce831-9151-418f-9430-231ad16fb538" containerName="payload-processing" Sep 14 10:18:11.106993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106984 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52ae09a-a490-4397-9a28-e0cadf6fe734" containerName="payload-processing" Sep 14 10:18:11.106993 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.106989 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52ae09a-a490-4397-9a28-e0cadf6fe734" containerName="payload-processing" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107003 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" containerName="maas-api" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107008 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" containerName="maas-api" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107016 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c3d860d-6ef8-45f8-9cae-789f39846cdf" containerName="payload-pre-processing" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107022 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c3d860d-6ef8-45f8-9cae-789f39846cdf" containerName="payload-pre-processing" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107028 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea72747d-540a-46b9-b97e-ee30264aa2a8" containerName="payload-pre-processing" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107033 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea72747d-540a-46b9-b97e-ee30264aa2a8" containerName="payload-pre-processing" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107085 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6270b1-4100-4ebd-b5e1-32a9655c9056" containerName="maas-api" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107097 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae23d1f-b98e-4ea9-a2ab-2b30cff4bc8b" containerName="maas-api" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107105 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52ae09a-a490-4397-9a28-e0cadf6fe734" containerName="payload-processing" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107111 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="19989a8b-8523-4b16-90a2-db5a9b372f2e" containerName="payload-processing" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107116 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6bce831-9151-418f-9430-231ad16fb538" containerName="payload-processing" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107122 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c3d860d-6ef8-45f8-9cae-789f39846cdf" containerName="payload-pre-processing" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107127 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="563cac49-ae20-4251-8a35-1e7f0aea1846" containerName="maas-api" Sep 14 10:18:11.107362 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.107132 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea72747d-540a-46b9-b97e-ee30264aa2a8" containerName="payload-pre-processing" Sep 14 10:18:11.110018 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.110000 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm" Sep 14 10:18:11.112732 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.112703 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\"" Sep 14 10:18:11.118982 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.118960 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm"] Sep 14 10:18:11.142668 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.142628 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04607b95-4f8a-4b99-b10b-8a247ba272c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm\" (UID: \"04607b95-4f8a-4b99-b10b-8a247ba272c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm" Sep 14 10:18:11.243148 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.243115 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04607b95-4f8a-4b99-b10b-8a247ba272c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm\" (UID: \"04607b95-4f8a-4b99-b10b-8a247ba272c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm" Sep 14 10:18:11.243765 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.243743 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04607b95-4f8a-4b99-b10b-8a247ba272c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm\" (UID: \"04607b95-4f8a-4b99-b10b-8a247ba272c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm" Sep 14 10:18:11.421723 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.421681 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm" Sep 14 10:18:11.692753 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:11.692725 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm"] Sep 14 10:18:11.694959 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:18:11.694929 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04607b95_4f8a_4b99_b10b_8a247ba272c3.slice/crio-d265f802b4b966ca25a3b7952dda17e28800bf50d60a6bc981217992bf63577b WatchSource:0}: Error finding container d265f802b4b966ca25a3b7952dda17e28800bf50d60a6bc981217992bf63577b: Status 404 returned error can't find the container with id d265f802b4b966ca25a3b7952dda17e28800bf50d60a6bc981217992bf63577b Sep 14 10:18:12.674664 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:12.674558 2550 generic.go:356] "Generic (PLEG): container finished" podID="04607b95-4f8a-4b99-b10b-8a247ba272c3" containerID="63aecdd62728ffabb052788147721c4e90e350ca7ca089ac31d4da1b1597111b" exitCode=0 Sep 14 10:18:12.674664 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:12.674656 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm" event={"ID":"04607b95-4f8a-4b99-b10b-8a247ba272c3","Type":"ContainerDied","Data":"63aecdd62728ffabb052788147721c4e90e350ca7ca089ac31d4da1b1597111b"} Sep 14 10:18:12.675031 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:12.674692 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm" event={"ID":"04607b95-4f8a-4b99-b10b-8a247ba272c3","Type":"ContainerStarted","Data":"d265f802b4b966ca25a3b7952dda17e28800bf50d60a6bc981217992bf63577b"} Sep 14 10:18:13.813259 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:13.813230 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm" Sep 14 10:18:13.865797 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:13.865762 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04607b95-4f8a-4b99-b10b-8a247ba272c3-maas-api-service-ca\") pod \"04607b95-4f8a-4b99-b10b-8a247ba272c3\" (UID: \"04607b95-4f8a-4b99-b10b-8a247ba272c3\") " Sep 14 10:18:13.866166 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:13.866141 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04607b95-4f8a-4b99-b10b-8a247ba272c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04607b95-4f8a-4b99-b10b-8a247ba272c3" (UID: "04607b95-4f8a-4b99-b10b-8a247ba272c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:18:13.967106 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:13.967016 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04607b95-4f8a-4b99-b10b-8a247ba272c3-maas-api-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:14.685568 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:14.685539 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm" Sep 14 10:18:14.685568 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:14.685545 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm" event={"ID":"04607b95-4f8a-4b99-b10b-8a247ba272c3","Type":"ContainerDied","Data":"d265f802b4b966ca25a3b7952dda17e28800bf50d60a6bc981217992bf63577b"} Sep 14 10:18:14.685850 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:14.685590 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d265f802b4b966ca25a3b7952dda17e28800bf50d60a6bc981217992bf63577b" Sep 14 10:18:21.723489 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:21.723446 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk"] Sep 14 10:18:21.724093 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:21.723700 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" podUID="e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801" containerName="payload-processing" containerID="cri-o://6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0" gracePeriod=30 Sep 14 10:18:21.925312 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:21.925283 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm"] Sep 14 10:18:21.928823 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:21.928796 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4d054-vkzbm"] Sep 14 10:18:21.982251 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:21.982197 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:18:22.040756 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.040723 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-plugins-config-volume\") pod \"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801\" (UID: \"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801\") " Sep 14 10:18:22.040917 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.040810 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f2cl\" (UniqueName: \"kubernetes.io/projected/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-kube-api-access-4f2cl\") pod \"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801\" (UID: \"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801\") " Sep 14 10:18:22.041177 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.041155 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801" (UID: "e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:18:22.042854 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.042830 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-kube-api-access-4f2cl" (OuterVolumeSpecName: "kube-api-access-4f2cl") pod "e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801" (UID: "e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801"). InnerVolumeSpecName "kube-api-access-4f2cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:18:22.142396 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.142365 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-plugins-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:22.142396 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.142392 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f2cl\" (UniqueName: \"kubernetes.io/projected/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801-kube-api-access-4f2cl\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:18:22.729735 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.729699 2550 generic.go:356] "Generic (PLEG): container finished" podID="e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801" containerID="6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0" exitCode=0 Sep 14 10:18:22.730165 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.729745 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" Sep 14 10:18:22.730165 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.729789 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" event={"ID":"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801","Type":"ContainerDied","Data":"6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0"} Sep 14 10:18:22.730165 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.729845 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk" event={"ID":"e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801","Type":"ContainerDied","Data":"fd79763f79cd75d5436e4e58e5840c8f4d571b3daabbd9c0bb06d91eba2799c5"} Sep 14 10:18:22.730165 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.729874 2550 scope.go:117] "RemoveContainer" containerID="6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0" Sep 14 10:18:22.739399 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.739344 2550 scope.go:117] "RemoveContainer" containerID="6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0" Sep 14 10:18:22.740180 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:18:22.739789 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0\": container with ID starting with 6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0 not found: ID does not exist" containerID="6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0" Sep 14 10:18:22.740180 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.739834 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0"} err="failed to get container status \"6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0\": rpc error: code = NotFound desc = could not find container \"6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0\": container with ID starting with 6c8256d96e0f0c08fb9ac36a30b08b87bbba43528c152f86d06395ec34ec88b0 not found: ID does not exist" Sep 14 10:18:22.754354 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.754310 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk"] Sep 14 10:18:22.756271 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:22.756251 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4d054-6dbff6c66c-xdjpk"] Sep 14 10:18:23.824219 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:23.824186 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04607b95-4f8a-4b99-b10b-8a247ba272c3" path="/var/lib/kubelet/pods/04607b95-4f8a-4b99-b10b-8a247ba272c3/volumes" Sep 14 10:18:23.824609 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:18:23.824566 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801" path="/var/lib/kubelet/pods/e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801/volumes" Sep 14 10:21:04.747838 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:04.747806 2550 scope.go:117] "RemoveContainer" containerID="ce9e6acc4822530b0092dd1902d1a6a5be846fa04a1206282b96d42503922ce1" Sep 14 10:21:07.522512 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:07.522482 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs"] Sep 14 10:21:07.522977 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:07.522692 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" podUID="b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" containerName="manager" containerID="cri-o://05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c" gracePeriod=10 Sep 14 10:21:08.181803 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.181776 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 10:21:08.222380 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.222345 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-extensions-socket-volume\") pod \"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9\" (UID: \"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9\") " Sep 14 10:21:08.222541 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.222447 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzvbz\" (UniqueName: \"kubernetes.io/projected/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-kube-api-access-kzvbz\") pod \"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9\" (UID: \"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9\") " Sep 14 10:21:08.222780 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.222754 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" (UID: "b8596cfd-3e6f-48e6-80be-e21ec8fc34d9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 10:21:08.224902 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.224874 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-kube-api-access-kzvbz" (OuterVolumeSpecName: "kube-api-access-kzvbz") pod "b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" (UID: "b8596cfd-3e6f-48e6-80be-e21ec8fc34d9"). InnerVolumeSpecName "kube-api-access-kzvbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:21:08.323898 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.323815 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzvbz\" (UniqueName: \"kubernetes.io/projected/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-kube-api-access-kzvbz\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:21:08.323898 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.323851 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9-extensions-socket-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:21:08.543422 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.543387 2550 generic.go:356] "Generic (PLEG): container finished" podID="b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" containerID="05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c" exitCode=0 Sep 14 10:21:08.543871 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.543431 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" Sep 14 10:21:08.543871 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.543453 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" event={"ID":"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9","Type":"ContainerDied","Data":"05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c"} Sep 14 10:21:08.543871 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.543517 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs" event={"ID":"b8596cfd-3e6f-48e6-80be-e21ec8fc34d9","Type":"ContainerDied","Data":"6f1e5bafc4dbf563f4608063cbe63abdcf3efbb83c48ad24985250f5e59c5679"} Sep 14 10:21:08.543871 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.543538 2550 scope.go:117] "RemoveContainer" containerID="05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c" Sep 14 10:21:08.552590 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.552566 2550 scope.go:117] "RemoveContainer" containerID="a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4" Sep 14 10:21:08.560202 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.560186 2550 scope.go:117] "RemoveContainer" containerID="05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c" Sep 14 10:21:08.560428 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:21:08.560409 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c\": container with ID starting with 05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c not found: ID does not exist" containerID="05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c" Sep 14 10:21:08.560478 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.560436 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c"} err="failed to get container status \"05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c\": rpc error: code = NotFound desc = could not find container \"05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c\": container with ID starting with 05f253fd4f9348d592cdc449481579053a7f2f1737666209f247449d023b354c not found: ID does not exist" Sep 14 10:21:08.560478 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.560457 2550 scope.go:117] "RemoveContainer" containerID="a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4" Sep 14 10:21:08.560707 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:21:08.560687 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4\": container with ID starting with a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4 not found: ID does not exist" containerID="a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4" Sep 14 10:21:08.560810 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.560708 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4"} err="failed to get container status \"a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4\": rpc error: code = NotFound desc = could not find container \"a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4\": container with ID starting with a04eff78cefb5777fbe1a864c2efa1f5a877aaaa3b71019e6f659481f92256e4 not found: ID does not exist" Sep 14 10:21:08.567125 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.567104 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs"] Sep 14 10:21:08.570328 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:08.570306 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k95fs"] Sep 14 10:21:09.824074 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:21:09.824040 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" path="/var/lib/kubelet/pods/b8596cfd-3e6f-48e6-80be-e21ec8fc34d9/volumes" Sep 14 10:22:08.265489 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265459 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m"] Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265869 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" containerName="manager" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265881 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" containerName="manager" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265894 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" containerName="manager" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265900 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" containerName="manager" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265906 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04607b95-4f8a-4b99-b10b-8a247ba272c3" containerName="revoke-keys" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265913 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="04607b95-4f8a-4b99-b10b-8a247ba272c3" containerName="revoke-keys" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265922 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801" containerName="payload-processing" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265928 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801" containerName="payload-processing" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265983 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" containerName="manager" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.265991 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69e38b7-f4e0-4ae0-9b46-ae6e1de9b801" containerName="payload-processing" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.266000 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="04607b95-4f8a-4b99-b10b-8a247ba272c3" containerName="revoke-keys" Sep 14 10:22:08.266006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.266007 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8596cfd-3e6f-48e6-80be-e21ec8fc34d9" containerName="manager" Sep 14 10:22:08.270205 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.270186 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" Sep 14 10:22:08.274238 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.274209 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zdclx\"" Sep 14 10:22:08.274831 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.274810 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 14 10:22:08.275577 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.275562 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 14 10:22:08.293892 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.293867 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m"] Sep 14 10:22:08.358047 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.358013 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00cadc2b-5289-401d-9307-80bff79fb3bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cc94m\" (UID: \"00cadc2b-5289-401d-9307-80bff79fb3bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" Sep 14 10:22:08.358207 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.358151 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcnhs\" (UniqueName: \"kubernetes.io/projected/00cadc2b-5289-401d-9307-80bff79fb3bb-kube-api-access-tcnhs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cc94m\" (UID: \"00cadc2b-5289-401d-9307-80bff79fb3bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" Sep 14 10:22:08.458816 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.458784 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00cadc2b-5289-401d-9307-80bff79fb3bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cc94m\" (UID: \"00cadc2b-5289-401d-9307-80bff79fb3bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" Sep 14 10:22:08.458952 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.458872 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcnhs\" (UniqueName: \"kubernetes.io/projected/00cadc2b-5289-401d-9307-80bff79fb3bb-kube-api-access-tcnhs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cc94m\" (UID: \"00cadc2b-5289-401d-9307-80bff79fb3bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" Sep 14 10:22:08.459145 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.459125 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00cadc2b-5289-401d-9307-80bff79fb3bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cc94m\" (UID: \"00cadc2b-5289-401d-9307-80bff79fb3bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" Sep 14 10:22:08.470344 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.470319 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcnhs\" (UniqueName: \"kubernetes.io/projected/00cadc2b-5289-401d-9307-80bff79fb3bb-kube-api-access-tcnhs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cc94m\" (UID: \"00cadc2b-5289-401d-9307-80bff79fb3bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" Sep 14 10:22:08.582145 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.582068 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" Sep 14 10:22:08.859359 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:08.859329 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m"] Sep 14 10:22:08.861260 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:22:08.861230 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cadc2b_5289_401d_9307_80bff79fb3bb.slice/crio-4b162a2424a028431552c1ad78e7405a018b7616e61badf46e3d1376db3cf353 WatchSource:0}: Error finding container 4b162a2424a028431552c1ad78e7405a018b7616e61badf46e3d1376db3cf353: Status 404 returned error can't find the container with id 4b162a2424a028431552c1ad78e7405a018b7616e61badf46e3d1376db3cf353 Sep 14 10:22:09.838528 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:09.838485 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" event={"ID":"00cadc2b-5289-401d-9307-80bff79fb3bb","Type":"ContainerStarted","Data":"23b37e05f5ac9e579004ee6d80abcadb73193d75ef3ed3d60c7c8c5a078e637c"} Sep 14 10:22:09.838952 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:09.838552 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" Sep 14 10:22:09.838952 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:09.838578 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" event={"ID":"00cadc2b-5289-401d-9307-80bff79fb3bb","Type":"ContainerStarted","Data":"4b162a2424a028431552c1ad78e7405a018b7616e61badf46e3d1376db3cf353"} Sep 14 10:22:09.860736 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:09.860687 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" podStartSLOduration=1.860673898 podStartE2EDuration="1.860673898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:22:09.858350585 +0000 UTC m=+2346.553475889" watchObservedRunningTime="2026-09-14 10:22:09.860673898 +0000 UTC m=+2346.555799203" Sep 14 10:22:20.845771 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:22:20.845741 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cc94m" Sep 14 10:23:04.029944 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:04.029919 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:23:04.038509 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:04.038483 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:23:47.590088 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.590052 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr"] Sep 14 10:23:47.593413 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.593397 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.598084 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.598061 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-87zkx\"" Sep 14 10:23:47.598244 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.598083 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 14 10:23:47.598244 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.598122 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 10:23:47.608211 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.608184 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr"] Sep 14 10:23:47.617760 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.617730 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq"] Sep 14 10:23:47.621814 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.621793 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.634577 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.634551 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq"] Sep 14 10:23:47.747865 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.747838 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-metrics-certs\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.747865 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.747870 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhpgl\" (UniqueName: \"kubernetes.io/projected/874e3df3-2112-425c-8b44-f96e1602320c-kube-api-access-fhpgl\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.748074 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.747903 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-maas-api-tls\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.748074 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.747935 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sj5j\" (UniqueName: \"kubernetes.io/projected/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-kube-api-access-5sj5j\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.748074 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.747966 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-maas-api-tls\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.748074 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.748000 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/874e3df3-2112-425c-8b44-f96e1602320c-openshift-service-ca\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.748074 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.748020 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-metrics-certs\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.748237 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.748101 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-openshift-service-ca\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.849415 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.849322 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sj5j\" (UniqueName: \"kubernetes.io/projected/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-kube-api-access-5sj5j\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.849415 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.849375 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-maas-api-tls\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.849415 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.849407 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/874e3df3-2112-425c-8b44-f96e1602320c-openshift-service-ca\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.849752 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.849434 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-metrics-certs\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.849752 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.849496 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-openshift-service-ca\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.849752 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.849540 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-metrics-certs\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.849752 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.849565 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhpgl\" (UniqueName: \"kubernetes.io/projected/874e3df3-2112-425c-8b44-f96e1602320c-kube-api-access-fhpgl\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.849752 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.849605 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-maas-api-tls\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.850308 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.850270 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/874e3df3-2112-425c-8b44-f96e1602320c-openshift-service-ca\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.850435 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.850398 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-openshift-service-ca\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.852072 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.852050 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-metrics-certs\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.852199 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.852092 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-maas-api-tls\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.852355 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.852335 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-maas-api-tls\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.852419 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.852386 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-metrics-certs\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.857882 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.857852 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhpgl\" (UniqueName: \"kubernetes.io/projected/874e3df3-2112-425c-8b44-f96e1602320c-kube-api-access-fhpgl\") pod \"maas-api-769969ccd5-mfxhr\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.858335 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.858319 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sj5j\" (UniqueName: \"kubernetes.io/projected/04ab38bd-cd7f-4d66-ad51-624b7ca11d49-kube-api-access-5sj5j\") pod \"maas-api-88c4c66b5-vmfvq\" (UID: \"04ab38bd-cd7f-4d66-ad51-624b7ca11d49\") " pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:47.904254 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.904218 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:47.936272 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:47.936237 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:48.186416 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:48.186392 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr"] Sep 14 10:23:48.188139 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:23:48.188118 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874e3df3_2112_425c_8b44_f96e1602320c.slice/crio-f8dfc94f789513704f9012f1ac06c4be534545bc074a93b06d3609f5cb45b310 WatchSource:0}: Error finding container f8dfc94f789513704f9012f1ac06c4be534545bc074a93b06d3609f5cb45b310: Status 404 returned error can't find the container with id f8dfc94f789513704f9012f1ac06c4be534545bc074a93b06d3609f5cb45b310 Sep 14 10:23:48.189494 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:48.189477 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 10:23:48.238157 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:48.238136 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq"] Sep 14 10:23:48.240180 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:23:48.240152 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ab38bd_cd7f_4d66_ad51_624b7ca11d49.slice/crio-ac993542cdd4b6307360f8f1eabfe645926d12b14d5f4c1a8c3a9d2c220f38ec WatchSource:0}: Error finding container ac993542cdd4b6307360f8f1eabfe645926d12b14d5f4c1a8c3a9d2c220f38ec: Status 404 returned error can't find the container with id ac993542cdd4b6307360f8f1eabfe645926d12b14d5f4c1a8c3a9d2c220f38ec Sep 14 10:23:48.311177 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:48.311127 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" event={"ID":"04ab38bd-cd7f-4d66-ad51-624b7ca11d49","Type":"ContainerStarted","Data":"ac993542cdd4b6307360f8f1eabfe645926d12b14d5f4c1a8c3a9d2c220f38ec"} Sep 14 10:23:48.312248 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:48.312221 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" event={"ID":"874e3df3-2112-425c-8b44-f96e1602320c","Type":"ContainerStarted","Data":"f8dfc94f789513704f9012f1ac06c4be534545bc074a93b06d3609f5cb45b310"} Sep 14 10:23:49.319263 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:49.319208 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" event={"ID":"04ab38bd-cd7f-4d66-ad51-624b7ca11d49","Type":"ContainerStarted","Data":"a8bef32c7568f22315d47ae212ec272fd7bfe38aece472c4cc0924d2fe25bf55"} Sep 14 10:23:49.319769 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:49.319282 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:49.320872 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:49.320847 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" event={"ID":"874e3df3-2112-425c-8b44-f96e1602320c","Type":"ContainerStarted","Data":"880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53"} Sep 14 10:23:49.320984 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:49.320900 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:49.365571 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:49.365517 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" podStartSLOduration=1.9065123339999999 podStartE2EDuration="2.365497704s" totalImagesPullingTime="458.98537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:23:49.342309781 +0000 UTC m=+2446.037435098" watchObservedRunningTime="2026-09-14 10:23:49.365497704 +0000 UTC m=+2446.060623010" Sep 14 10:23:49.366080 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:49.366045 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" podStartSLOduration=1.905508898 podStartE2EDuration="2.366038904s" totalImagesPullingTime="460.530006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:23:49.363809075 +0000 UTC m=+2446.058934380" watchObservedRunningTime="2026-09-14 10:23:49.366038904 +0000 UTC m=+2446.061164209" Sep 14 10:23:55.332687 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.332655 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-88c4c66b5-vmfvq" Sep 14 10:23:55.333112 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.332730 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:55.413726 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.413691 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr"] Sep 14 10:23:55.413923 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.413902 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" podUID="874e3df3-2112-425c-8b44-f96e1602320c" containerName="maas-api" containerID="cri-o://880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53" gracePeriod=30 Sep 14 10:23:55.667285 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.667262 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:55.725481 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.725446 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhpgl\" (UniqueName: \"kubernetes.io/projected/874e3df3-2112-425c-8b44-f96e1602320c-kube-api-access-fhpgl\") pod \"874e3df3-2112-425c-8b44-f96e1602320c\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " Sep 14 10:23:55.725658 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.725499 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/874e3df3-2112-425c-8b44-f96e1602320c-openshift-service-ca\") pod \"874e3df3-2112-425c-8b44-f96e1602320c\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " Sep 14 10:23:55.725658 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.725530 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-maas-api-tls\") pod \"874e3df3-2112-425c-8b44-f96e1602320c\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " Sep 14 10:23:55.725658 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.725572 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-metrics-certs\") pod \"874e3df3-2112-425c-8b44-f96e1602320c\" (UID: \"874e3df3-2112-425c-8b44-f96e1602320c\") " Sep 14 10:23:55.725958 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.725930 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874e3df3-2112-425c-8b44-f96e1602320c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "874e3df3-2112-425c-8b44-f96e1602320c" (UID: "874e3df3-2112-425c-8b44-f96e1602320c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:23:55.727765 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.727719 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "874e3df3-2112-425c-8b44-f96e1602320c" (UID: "874e3df3-2112-425c-8b44-f96e1602320c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:23:55.727884 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.727763 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874e3df3-2112-425c-8b44-f96e1602320c-kube-api-access-fhpgl" (OuterVolumeSpecName: "kube-api-access-fhpgl") pod "874e3df3-2112-425c-8b44-f96e1602320c" (UID: "874e3df3-2112-425c-8b44-f96e1602320c"). InnerVolumeSpecName "kube-api-access-fhpgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:23:55.727884 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.727773 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "874e3df3-2112-425c-8b44-f96e1602320c" (UID: "874e3df3-2112-425c-8b44-f96e1602320c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:23:55.826202 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.826169 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/874e3df3-2112-425c-8b44-f96e1602320c-openshift-service-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:23:55.826202 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.826194 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-maas-api-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:23:55.826202 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.826203 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/874e3df3-2112-425c-8b44-f96e1602320c-metrics-certs\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:23:55.826202 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:55.826211 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhpgl\" (UniqueName: \"kubernetes.io/projected/874e3df3-2112-425c-8b44-f96e1602320c-kube-api-access-fhpgl\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 14 10:23:56.357996 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:56.357964 2550 generic.go:356] "Generic (PLEG): container finished" podID="874e3df3-2112-425c-8b44-f96e1602320c" containerID="880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53" exitCode=0 Sep 14 10:23:56.358402 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:56.358010 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" Sep 14 10:23:56.358402 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:56.358016 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" event={"ID":"874e3df3-2112-425c-8b44-f96e1602320c","Type":"ContainerDied","Data":"880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53"} Sep 14 10:23:56.358402 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:56.358056 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr" event={"ID":"874e3df3-2112-425c-8b44-f96e1602320c","Type":"ContainerDied","Data":"f8dfc94f789513704f9012f1ac06c4be534545bc074a93b06d3609f5cb45b310"} Sep 14 10:23:56.358402 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:56.358077 2550 scope.go:117] "RemoveContainer" containerID="880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53" Sep 14 10:23:56.366802 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:56.366785 2550 scope.go:117] "RemoveContainer" containerID="880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53" Sep 14 10:23:56.367067 ip-10-0-135-40 kubenswrapper[2550]: E0914 10:23:56.367048 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53\": container with ID starting with 880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53 not found: ID does not exist" containerID="880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53" Sep 14 10:23:56.367120 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:56.367076 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53"} err="failed to get container status \"880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53\": rpc error: code = NotFound desc = could not find container \"880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53\": container with ID starting with 880144a2bccaa73f2edb41abe123478f9920c974fef983810794126d23c21a53 not found: ID does not exist" Sep 14 10:23:56.377006 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:56.376983 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr"] Sep 14 10:23:56.382140 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:56.382118 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-769969ccd5-mfxhr"] Sep 14 10:23:57.826040 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:23:57.826004 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="874e3df3-2112-425c-8b44-f96e1602320c" path="/var/lib/kubelet/pods/874e3df3-2112-425c-8b44-f96e1602320c/volumes" Sep 14 10:24:23.077812 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:23.077779 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-l5m7r_b2224a72-31fa-4624-ae8c-29fef205f2ee/manager/1.log" Sep 14 10:24:23.440183 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:23.440152 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fdd9869c-kz6d7_0b42aeb4-6c44-485b-9129-8c64cad0a11a/manager/0.log" Sep 14 10:24:23.465974 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:23.465948 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fdd9869c-kz6d7_0b42aeb4-6c44-485b-9129-8c64cad0a11a/copy-manifests/0.log" Sep 14 10:24:23.806905 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:23.806823 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-88c4c66b5-vmfvq_04ab38bd-cd7f-4d66-ad51-624b7ca11d49/maas-api/0.log" Sep 14 10:24:24.256282 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:24.256249 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-plj7l_05274f1c-f432-455a-96fd-36f1f2bad255/postgres/0.log" Sep 14 10:24:25.489317 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:25.489284 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-cc94m_00cadc2b-5289-401d-9307-80bff79fb3bb/manager/0.log" Sep 14 10:24:26.207311 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:26.207273 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-md4f7_b568871b-3bca-40c4-8ca3-c90b8cb94373/discovery/0.log" Sep 14 10:24:26.657256 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:26.657174 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dv48j_6e53b02d-3fed-459f-a481-c850ea69f9e1/payload-pre-processing/0.log" Sep 14 10:24:27.558308 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:27.558277 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ztn9g_9646de9f-9892-49b2-a276-930c859b9515/main/0.log" Sep 14 10:24:36.970902 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:36.970873 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z8prz_94f8bdb5-679a-4df1-9422-497f18ff1bbc/global-pull-secret-syncer/0.log" Sep 14 10:24:37.076344 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:37.076312 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gfw87_60182aa7-5db0-4cb3-b543-7b29dc4fafe3/konnectivity-agent/0.log" Sep 14 10:24:37.186318 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:37.186257 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-40.ec2.internal_98f5720cef50a512c0a0bcbc18cd38fe/haproxy/0.log" Sep 14 10:24:42.036113 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:42.036076 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-cc94m_00cadc2b-5289-401d-9307-80bff79fb3bb/manager/0.log" Sep 14 10:24:43.561074 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:43.561045 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7e632b34-a7bd-4f48-8750-bfdc1eca6b9e/alertmanager/0.log" Sep 14 10:24:43.592458 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:43.592421 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7e632b34-a7bd-4f48-8750-bfdc1eca6b9e/config-reloader/0.log" Sep 14 10:24:43.620807 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:43.620782 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7e632b34-a7bd-4f48-8750-bfdc1eca6b9e/kube-rbac-proxy-web/0.log" Sep 14 10:24:43.647591 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:43.647565 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7e632b34-a7bd-4f48-8750-bfdc1eca6b9e/kube-rbac-proxy/0.log" Sep 14 10:24:43.682190 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:43.682167 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7e632b34-a7bd-4f48-8750-bfdc1eca6b9e/kube-rbac-proxy-metric/0.log" Sep 14 10:24:43.711595 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:43.711565 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7e632b34-a7bd-4f48-8750-bfdc1eca6b9e/prom-label-proxy/0.log" Sep 14 10:24:43.737608 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:43.737583 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7e632b34-a7bd-4f48-8750-bfdc1eca6b9e/init-config-reloader/0.log" Sep 14 10:24:44.009420 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.009392 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7j5mh_779e293c-f1ff-48c9-9f68-bfaae2520712/node-exporter/0.log" Sep 14 10:24:44.036675 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.036645 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7j5mh_779e293c-f1ff-48c9-9f68-bfaae2520712/kube-rbac-proxy/0.log" Sep 14 10:24:44.064285 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.064258 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7j5mh_779e293c-f1ff-48c9-9f68-bfaae2520712/init-textfile/0.log" Sep 14 10:24:44.425725 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.425687 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36/prometheus/0.log" Sep 14 10:24:44.454187 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.454157 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36/config-reloader/0.log" Sep 14 10:24:44.483491 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.483457 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36/thanos-sidecar/0.log" Sep 14 10:24:44.508941 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.508918 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36/kube-rbac-proxy-web/0.log" Sep 14 10:24:44.536077 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.536047 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36/kube-rbac-proxy/0.log" Sep 14 10:24:44.560590 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.560563 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36/kube-rbac-proxy-thanos/0.log" Sep 14 10:24:44.586827 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.586802 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1b4013d-ad1a-4d7b-8d41-66bfaadfbe36/init-config-reloader/0.log" Sep 14 10:24:44.685499 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.685471 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-48lrj_94a2a535-c7dd-40f8-a620-67aedbbabbe3/prometheus-operator-admission-webhook/0.log" Sep 14 10:24:44.723094 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.723067 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5486df5f55-kj7bk_b3f931a3-7a96-44d7-beac-43811b45bb1a/telemeter-client/0.log" Sep 14 10:24:44.750918 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.750895 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5486df5f55-kj7bk_b3f931a3-7a96-44d7-beac-43811b45bb1a/reload/0.log" Sep 14 10:24:44.776135 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.776113 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5486df5f55-kj7bk_b3f931a3-7a96-44d7-beac-43811b45bb1a/kube-rbac-proxy/0.log" Sep 14 10:24:44.818573 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.818556 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd55dd6f9-f5vrb_feb0e927-0450-41ab-9427-93dfe0392abe/thanos-query/0.log" Sep 14 10:24:44.844331 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.844314 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd55dd6f9-f5vrb_feb0e927-0450-41ab-9427-93dfe0392abe/kube-rbac-proxy-web/0.log" Sep 14 10:24:44.870043 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.870021 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd55dd6f9-f5vrb_feb0e927-0450-41ab-9427-93dfe0392abe/kube-rbac-proxy/0.log" Sep 14 10:24:44.896473 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.896456 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd55dd6f9-f5vrb_feb0e927-0450-41ab-9427-93dfe0392abe/prom-label-proxy/0.log" Sep 14 10:24:44.929379 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.929360 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd55dd6f9-f5vrb_feb0e927-0450-41ab-9427-93dfe0392abe/kube-rbac-proxy-rules/0.log" Sep 14 10:24:44.954122 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:44.954070 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd55dd6f9-f5vrb_feb0e927-0450-41ab-9427-93dfe0392abe/kube-rbac-proxy-metrics/0.log" Sep 14 10:24:46.961603 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:46.961567 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/1.log" Sep 14 10:24:46.969962 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:46.969932 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-76v9c_6a2690c3-8c2f-4124-8575-e75a9a44d107/console-operator/2.log" Sep 14 10:24:47.023676 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.023648 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88"] Sep 14 10:24:47.024151 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.024041 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="874e3df3-2112-425c-8b44-f96e1602320c" containerName="maas-api" Sep 14 10:24:47.024151 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.024060 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="874e3df3-2112-425c-8b44-f96e1602320c" containerName="maas-api" Sep 14 10:24:47.024331 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.024168 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="874e3df3-2112-425c-8b44-f96e1602320c" containerName="maas-api" Sep 14 10:24:47.027424 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.027403 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.030237 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.030218 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhktf\"/\"kube-root-ca.crt\"" Sep 14 10:24:47.030331 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.030286 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhktf\"/\"default-dockercfg-7kn6x\"" Sep 14 10:24:47.031197 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.031153 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhktf\"/\"openshift-service-ca.crt\"" Sep 14 10:24:47.038747 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.038726 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88"] Sep 14 10:24:47.097964 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.097933 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clrh4\" (UniqueName: \"kubernetes.io/projected/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-kube-api-access-clrh4\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.098135 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.097970 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-sys\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.098135 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.097993 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-lib-modules\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.098135 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.098075 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-proc\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.098135 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.098114 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-podres\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.198786 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.198750 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-proc\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.198786 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.198788 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-podres\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.199007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.198880 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-proc\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.199007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.198921 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clrh4\" (UniqueName: \"kubernetes.io/projected/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-kube-api-access-clrh4\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.199007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.198945 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-sys\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.199007 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.198967 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-lib-modules\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.199142 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.199002 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-podres\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.199142 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.199023 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-sys\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.199142 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.199086 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-lib-modules\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.207318 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.207289 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clrh4\" (UniqueName: \"kubernetes.io/projected/059de5a5-aba0-430e-b0cd-c8b17a0a87f9-kube-api-access-clrh4\") pod \"perf-node-gather-daemonset-sxr88\" (UID: \"059de5a5-aba0-430e-b0cd-c8b17a0a87f9\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.337923 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.337835 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:47.613164 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.613142 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88"] Sep 14 10:24:47.615321 ip-10-0-135-40 kubenswrapper[2550]: W0914 10:24:47.615284 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod059de5a5_aba0_430e_b0cd_c8b17a0a87f9.slice/crio-d9087be24b9259fb7fb8eaf86c7288d801d3d6974e405d49e009c0edb83fe6d5 WatchSource:0}: Error finding container d9087be24b9259fb7fb8eaf86c7288d801d3d6974e405d49e009c0edb83fe6d5: Status 404 returned error can't find the container with id d9087be24b9259fb7fb8eaf86c7288d801d3d6974e405d49e009c0edb83fe6d5 Sep 14 10:24:47.681648 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:47.681603 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-6dhlj_9ca5a196-fcb6-47d1-aef4-24eea70254c6/download-server/0.log" Sep 14 10:24:48.446330 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:48.446293 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-n8hq7_2d3ccc69-de69-4209-a388-ce7d1c1b45b9/volume-data-source-validator/0.log" Sep 14 10:24:48.610395 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:48.610356 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" event={"ID":"059de5a5-aba0-430e-b0cd-c8b17a0a87f9","Type":"ContainerStarted","Data":"f710c8449ceb7254c07b7141cc108e5783fd0470944d3988087fb34c926e1d93"} Sep 14 10:24:48.610802 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:48.610420 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" event={"ID":"059de5a5-aba0-430e-b0cd-c8b17a0a87f9","Type":"ContainerStarted","Data":"d9087be24b9259fb7fb8eaf86c7288d801d3d6974e405d49e009c0edb83fe6d5"} Sep 14 10:24:48.610802 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:48.610460 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:48.631262 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:48.631219 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" podStartSLOduration=1.631196834 podStartE2EDuration="1.631196834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:24:48.630104201 +0000 UTC m=+2505.325229505" watchObservedRunningTime="2026-09-14 10:24:48.631196834 +0000 UTC m=+2505.326322139" Sep 14 10:24:49.729057 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:49.729028 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q5n48_a7e4ee1c-60de-43ff-9b68-507c7ff58bdb/dns/0.log" Sep 14 10:24:49.752720 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:49.752693 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q5n48_a7e4ee1c-60de-43ff-9b68-507c7ff58bdb/kube-rbac-proxy/0.log" Sep 14 10:24:49.778861 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:49.778834 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4htsc_eae3b65a-f2f9-49e5-83ce-6bb3e6cb57ed/dns-node-resolver/0.log" Sep 14 10:24:50.487121 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:50.487088 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-phdx8_52a488f7-cd8d-4974-b6cb-5e293a21a2a6/node-ca/0.log" Sep 14 10:24:52.322859 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:52.322831 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-md4f7_b568871b-3bca-40c4-8ca3-c90b8cb94373/discovery/0.log" Sep 14 10:24:52.459844 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:52.459815 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dv48j_6e53b02d-3fed-459f-a481-c850ea69f9e1/payload-pre-processing/0.log" Sep 14 10:24:53.457500 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:53.457469 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-85622_53089d2e-4275-4a32-96e6-5800b3f71b08/serve-healthcheck-canary/0.log" Sep 14 10:24:54.132848 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:54.132815 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tln2x_0607799d-9921-4e2d-ad78-c4816e060358/kube-rbac-proxy/0.log" Sep 14 10:24:54.159194 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:54.159163 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tln2x_0607799d-9921-4e2d-ad78-c4816e060358/exporter/0.log" Sep 14 10:24:54.185207 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:54.185179 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tln2x_0607799d-9921-4e2d-ad78-c4816e060358/extractor/0.log" Sep 14 10:24:54.624882 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:54.624853 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-sxr88" Sep 14 10:24:56.921651 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:56.921603 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-l5m7r_b2224a72-31fa-4624-ae8c-29fef205f2ee/manager/0.log" Sep 14 10:24:56.968550 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:56.968518 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-l5m7r_b2224a72-31fa-4624-ae8c-29fef205f2ee/manager/1.log" Sep 14 10:24:57.210532 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:57.210501 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fdd9869c-kz6d7_0b42aeb4-6c44-485b-9129-8c64cad0a11a/manager/0.log" Sep 14 10:24:57.233521 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:57.233502 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fdd9869c-kz6d7_0b42aeb4-6c44-485b-9129-8c64cad0a11a/copy-manifests/0.log" Sep 14 10:24:58.688195 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:58.688158 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d6ddf947f-8bt8g_a1e73301-ae3f-4061-8410-ec2bba9d5941/manager/0.log" Sep 14 10:24:58.731141 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:58.731112 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-md4f7_b568871b-3bca-40c4-8ca3-c90b8cb94373/discovery/0.log" Sep 14 10:24:58.749646 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:58.749613 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-6pkc8_ce610de1-c41f-4a1d-8783-ba4b5e39c628/openshift-lws-operator/0.log" Sep 14 10:24:58.819140 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:24:58.819045 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dv48j_6e53b02d-3fed-459f-a481-c850ea69f9e1/payload-pre-processing/0.log" Sep 14 10:25:01.249220 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:25:01.249183 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-md4f7_b568871b-3bca-40c4-8ca3-c90b8cb94373/discovery/0.log" Sep 14 10:25:01.346755 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:25:01.346725 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dv48j_6e53b02d-3fed-459f-a481-c850ea69f9e1/payload-pre-processing/0.log" Sep 14 10:25:04.799215 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:25:04.799184 2550 scope.go:117] "RemoveContainer" containerID="63aecdd62728ffabb052788147721c4e90e350ca7ca089ac31d4da1b1597111b" Sep 14 10:25:05.645153 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:25:05.645126 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59vql_b4a830d4-0f56-4eb4-a2d7-e967a5a22256/kube-multus-additional-cni-plugins/0.log" Sep 14 10:25:05.668129 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:25:05.668100 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59vql_b4a830d4-0f56-4eb4-a2d7-e967a5a22256/egress-router-binary-copy/0.log" Sep 14 10:25:05.692074 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:25:05.692042 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59vql_b4a830d4-0f56-4eb4-a2d7-e967a5a22256/cni-plugins/0.log" Sep 14 10:25:05.714565 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:25:05.714539 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59vql_b4a830d4-0f56-4eb4-a2d7-e967a5a22256/bond-cni-plugin/0.log" Sep 14 10:25:05.736873 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:25:05.736850 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59vql_b4a830d4-0f56-4eb4-a2d7-e967a5a22256/routeoverride-cni/0.log" Sep 14 10:25:05.760259 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:25:05.760209 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59vql_b4a830d4-0f56-4eb4-a2d7-e967a5a22256/whereabouts-cni-bincopy/0.log" Sep 14 10:25:05.784232 ip-10-0-135-40 kubenswrapper[2550]: I0914 10:25:05.784209 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59vql_b4a830d4-0f56-4eb4-a2d7-e967a5a22256/whereabouts-cni/0.log"