Sep 16 07:31:55.112878 ip-10-0-131-134 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 07:31:55.112888 ip-10-0-131-134 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 07:31:55.112896 ip-10-0-131-134 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 07:31:55.113160 ip-10-0-131-134 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 07:32:05.361577 ip-10-0-131-134 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 07:32:05.361596 ip-10-0-131-134 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9fdfb68c9c754047902d36a4ff830582 -- Sep 16 07:33:46.977526 ip-10-0-131-134 systemd[1]: Starting Kubernetes Kubelet... Sep 16 07:33:47.328553 ip-10-0-131-134 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 07:33:47.328553 ip-10-0-131-134 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 07:33:47.328553 ip-10-0-131-134 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 07:33:47.328553 ip-10-0-131-134 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 07:33:47.328553 ip-10-0-131-134 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 07:33:47.329641 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.329559 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 07:33:47.334671 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334652 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:47.334671 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334668 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:47.334671 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334675 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334682 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334687 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334691 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334695 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334699 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334703 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334706 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334711 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334715 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334718 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334722 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334726 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334730 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334734 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334738 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334742 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334747 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334751 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:47.334873 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334755 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334759 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334763 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334768 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334771 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334775 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334779 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334783 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334787 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334791 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334795 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334799 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334804 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334808 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334812 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334817 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334821 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334826 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334830 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334834 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:47.335666 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334848 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334853 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334857 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334862 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334866 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334870 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334874 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334878 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334882 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334886 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334891 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334895 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334900 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334905 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334910 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334915 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334919 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334927 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334932 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:47.336518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334937 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334942 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334946 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334951 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334955 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334960 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334964 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334968 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334973 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334978 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334982 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334986 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334991 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.334997 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335003 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335007 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335011 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335015 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335019 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335024 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:47.336997 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335028 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335032 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335036 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335041 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335045 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335049 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335053 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335057 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335062 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335066 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335073 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335078 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335083 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335088 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335093 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335097 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335101 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335106 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.335110 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336295 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:47.337518 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336305 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336310 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336315 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336319 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336324 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336328 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336335 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336343 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336348 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336352 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336357 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336362 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336366 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336370 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336374 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336378 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336383 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336387 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336391 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336395 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:47.338039 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336399 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336404 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336408 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336412 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336416 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336421 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336425 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336429 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336433 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336437 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336441 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336446 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336451 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336456 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336460 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336464 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336469 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336473 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336478 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336482 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:47.338553 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336488 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336492 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336497 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336501 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336505 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336509 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336514 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336518 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336523 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336527 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336531 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336535 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336539 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336543 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336572 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336594 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336600 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336603 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336607 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:47.339031 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336610 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336614 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336617 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336620 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336622 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336625 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336628 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336631 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336634 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336636 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336639 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336642 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336645 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336648 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336650 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336653 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336656 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336658 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336661 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336664 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:47.339509 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336666 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336670 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336673 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336675 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336681 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336684 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336686 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336689 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336694 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336697 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336700 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336703 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336705 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336708 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336711 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336714 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336716 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336719 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.336723 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337111 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:47.339993 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337117 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337120 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337123 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337126 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337129 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337132 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337135 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337138 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337141 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337144 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337147 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337150 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337152 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337155 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337158 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337161 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337165 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337167 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337170 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337175 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:47.340496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337179 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337183 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337185 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337188 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337191 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337194 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337196 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337199 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337201 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337204 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337206 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337222 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337225 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337228 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337230 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337233 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337236 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337238 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337241 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337243 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:47.340987 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337246 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337248 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337251 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337254 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337257 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337260 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337263 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337266 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337268 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337271 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337273 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337276 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337279 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337282 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337285 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337288 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337290 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337293 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337295 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337298 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:47.341508 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337300 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337302 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337305 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337307 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337310 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337313 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337316 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337318 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337321 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337323 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337326 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337328 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337331 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337333 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337336 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337338 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337341 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337344 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337346 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337349 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:47.341991 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337351 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337354 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337356 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337359 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337362 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337365 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337368 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337370 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337373 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337376 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337378 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337381 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337383 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337388 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337392 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337395 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337398 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.337402 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337482 2580 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337534 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 07:33:47.342548 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337541 2580 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337546 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337550 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337553 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337557 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337562 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337565 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337568 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337572 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337575 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337579 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337581 2580 flags.go:64] FLAG: --cgroup-root="" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337584 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337587 2580 flags.go:64] FLAG: --client-ca-file="" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337590 2580 flags.go:64] FLAG: --cloud-provider="external" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337593 2580 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337597 2580 flags.go:64] FLAG: --cluster-domain="" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337600 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337603 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337607 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337610 2580 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337614 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337617 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 07:33:47.343025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337620 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337624 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337627 2580 flags.go:64] FLAG: --contention-profiling="false" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337630 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337633 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337636 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337639 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337643 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337646 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337649 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337652 2580 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337655 2580 flags.go:64] FLAG: --enable-server="true" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337658 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337662 2580 flags.go:64] FLAG: --event-burst="100" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337665 2580 flags.go:64] FLAG: --event-qps="50" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337669 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337672 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337675 2580 flags.go:64] FLAG: --eviction-hard="" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337679 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337682 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337685 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337688 2580 flags.go:64] FLAG: --eviction-soft="" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337691 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337694 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337697 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 07:33:47.343599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337699 2580 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337702 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337705 2580 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337708 2580 flags.go:64] FLAG: --feature-gates="" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337712 2580 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337715 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337718 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337721 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337724 2580 flags.go:64] FLAG: --healthz-port="10248" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337728 2580 flags.go:64] FLAG: --help="false" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337730 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337733 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337736 2580 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337739 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337743 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337746 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337749 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337752 2580 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337754 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337757 2580 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337760 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337763 2580 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337766 2580 flags.go:64] FLAG: --kube-reserved="" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337769 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 07:33:47.344235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337772 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337775 2580 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337778 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337780 2580 flags.go:64] FLAG: --lock-file="" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337783 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337786 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337789 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337794 2580 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337800 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337802 2580 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337805 2580 flags.go:64] FLAG: --logging-format="text" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337808 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337811 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337815 2580 flags.go:64] FLAG: --manifest-url="" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337818 2580 flags.go:64] FLAG: --manifest-url-header="" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337822 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337825 2580 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337829 2580 flags.go:64] FLAG: --max-pods="110" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337832 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337835 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337838 2580 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337841 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337844 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337847 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337850 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 07:33:47.344833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337857 2580 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337860 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337863 2580 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337866 2580 flags.go:64] FLAG: --pod-cidr="" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337869 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337874 2580 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337877 2580 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337880 2580 flags.go:64] FLAG: --pods-per-core="0" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337883 2580 flags.go:64] FLAG: --port="10250" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337886 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337889 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04abcb66da6ff8574" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337892 2580 flags.go:64] FLAG: --qos-reserved="" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337895 2580 flags.go:64] FLAG: --read-only-port="10255" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337898 2580 flags.go:64] FLAG: --register-node="true" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337903 2580 flags.go:64] FLAG: --register-with-taints="" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337906 2580 flags.go:64] FLAG: --registry-burst="10" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337911 2580 flags.go:64] FLAG: --registry-qps="5" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337918 2580 flags.go:64] FLAG: --reserved-cpus="" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337921 2580 flags.go:64] FLAG: --reserved-memory="" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337925 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337928 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337932 2580 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337935 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337937 2580 flags.go:64] FLAG: --runonce="false" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337940 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 07:33:47.345758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337943 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337946 2580 flags.go:64] FLAG: --seccomp-default="false" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337949 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337952 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337955 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337958 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337961 2580 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337964 2580 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337967 2580 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337970 2580 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337973 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337976 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337979 2580 flags.go:64] FLAG: --system-cgroups="" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337981 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337987 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337990 2580 flags.go:64] FLAG: --tls-cert-file="" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337993 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337996 2580 flags.go:64] FLAG: --tls-min-version="" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.337999 2580 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.338002 2580 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.338005 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.338009 2580 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.338011 2580 flags.go:64] FLAG: --v="2" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.338018 2580 flags.go:64] FLAG: --version="false" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.338022 2580 flags.go:64] FLAG: --vmodule="" Sep 16 07:33:47.346542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.338026 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.338029 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338137 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338141 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338146 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338148 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338151 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338154 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338156 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338160 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338162 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338165 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338167 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338171 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338185 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338188 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338191 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338194 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338197 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338200 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:47.347134 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338203 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338206 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338220 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338224 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338226 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338229 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338232 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338234 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338240 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338244 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338248 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338252 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338255 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338258 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338261 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338264 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338267 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338270 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338272 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:47.347729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338275 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338277 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338280 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338283 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338285 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338288 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338290 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338293 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338296 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338298 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338301 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338304 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338306 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338309 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338313 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338317 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338319 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338322 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338325 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338328 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:47.348184 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338330 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338334 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338337 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338341 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338343 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338346 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338349 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338352 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338354 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338357 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338359 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338363 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338366 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338368 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338371 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338373 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338376 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338378 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338381 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338384 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:47.348686 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338386 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338389 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338392 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338394 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338397 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338399 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338402 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338404 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338407 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338409 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338412 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338415 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338418 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338422 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338425 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338429 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338432 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338434 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338437 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338439 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:47.349173 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338442 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.338445 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.338988 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.347452 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.347470 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.348394 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.348405 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348465 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348470 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348474 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348477 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348480 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348484 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348487 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348489 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:47.349689 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348492 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348495 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348498 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348500 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348504 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348509 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348512 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348515 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348517 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348521 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348526 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348529 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348532 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348536 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348538 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348541 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348544 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348547 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348553 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:47.350094 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348556 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348558 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348561 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348564 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348566 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348569 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348571 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348574 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348577 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348579 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348582 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348584 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348587 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348589 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348594 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348597 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348600 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348603 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348606 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348609 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:47.350562 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348612 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348615 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348618 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348620 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348623 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348626 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348628 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348631 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348634 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348637 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348640 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348642 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348645 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348647 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348650 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348653 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348656 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348658 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348661 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348663 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:47.351042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348666 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348668 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348671 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348674 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348676 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348679 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348681 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348684 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348687 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348690 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348692 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348695 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348699 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348702 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348705 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348707 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348710 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348713 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348715 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348718 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:47.351655 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348721 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348723 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348726 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348728 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348731 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348734 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348736 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348739 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348742 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348744 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348747 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348750 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.348755 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348866 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348870 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:47.352142 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348873 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348876 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348879 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348882 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348884 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348887 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348890 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348892 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348895 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348897 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348901 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348903 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348906 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348909 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348912 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348915 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348919 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348922 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348925 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:47.352520 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348928 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348931 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348934 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348937 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348940 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348943 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348945 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348948 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348950 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348953 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348955 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348958 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348960 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348963 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348965 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348968 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348971 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348973 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348976 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348978 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:47.352980 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348981 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348984 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348988 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348991 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348994 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.348997 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349000 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349002 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349005 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349008 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349011 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349013 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349016 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349018 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349021 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349024 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349026 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349029 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349031 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349034 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:47.353463 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349037 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349039 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349041 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349044 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349047 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349049 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349052 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349054 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349057 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349060 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349062 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349065 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349067 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349070 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349072 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349075 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349077 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349081 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349084 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:47.353958 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349087 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349093 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349096 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349099 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349102 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349105 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349108 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349110 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349113 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349116 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349118 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349121 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349124 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349126 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349129 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349131 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349134 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349137 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:47.354451 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:47.349139 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:47.354895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.349144 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 07:33:47.354895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.349265 2580 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 07:33:47.354895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.351377 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 07:33:47.354895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.352161 2580 server.go:1019] "Starting client certificate rotation" Sep 16 07:33:47.354895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.352239 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 07:33:47.354895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.352296 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 07:33:47.354895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.354431 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 07:33:47.356414 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.356396 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 07:33:47.365590 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.365574 2580 log.go:25] "Validated CRI v1 runtime API" Sep 16 07:33:47.369860 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.369839 2580 log.go:25] "Validated CRI v1 image API" Sep 16 07:33:47.370717 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.370700 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 07:33:47.373516 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.373488 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d716d547-41c3-4b6f-b488-673e154f8e81:/dev/nvme0n1p4 da42e774-14bf-4509-8916-a51300765851:/dev/nvme0n1p3] Sep 16 07:33:47.373577 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.373516 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 07:33:47.379031 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.378922 2580 manager.go:216] Machine: {Timestamp:2026-09-16 07:33:47.377737903 +0000 UTC m=+0.297353839 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104263 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec217559fd80d9e953d4e2966db6e334 SystemUUID:ec217559-fd80-d9e9-53d4-e2966db6e334 BootID:9fdfb68c-9c75-4047-902d-36a4ff830582 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d6:eb:54:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d6:eb:54:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:1a:44:5f:73:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 07:33:47.379031 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.379019 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 07:33:47.379158 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.379089 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 07:33:47.379633 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.379617 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 07:33:47.382337 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.382313 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 07:33:47.382467 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.382341 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 07:33:47.383112 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.383102 2580 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 07:33:47.383145 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.383114 2580 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 07:33:47.383145 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.383122 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 07:33:47.383712 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.383702 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 07:33:47.384392 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.384382 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 07:33:47.388043 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.388031 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 16 07:33:47.388158 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.388149 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 07:33:47.388191 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.388178 2580 kubelet.go:486] "Attempting to sync node with API server" Sep 16 07:33:47.388191 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.388190 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 07:33:47.388280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.388200 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 07:33:47.388280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.388229 2580 kubelet.go:398] "Adding apiserver pod source" Sep 16 07:33:47.388280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.388245 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 07:33:47.388647 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.388633 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 07:33:47.388683 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.388661 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 07:33:47.390512 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.390495 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 07:33:47.392094 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.392082 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 07:33:47.392144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.392112 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 07:33:47.394031 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394016 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 07:33:47.394073 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394040 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 07:33:47.394073 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394049 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 07:33:47.394073 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394054 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 07:33:47.394073 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394062 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 07:33:47.394073 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394068 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 07:33:47.394200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394082 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 07:33:47.394200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394087 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 07:33:47.394200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394093 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 07:33:47.394200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394099 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 07:33:47.394200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394107 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 07:33:47.394200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394112 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 07:33:47.394760 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394746 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 07:33:47.394808 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.394761 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 07:33:47.398332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.398315 2580 server.go:1268] "Started kubelet" Sep 16 07:33:47.398427 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.398395 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 07:33:47.398503 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.398450 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 07:33:47.398551 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.398528 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 07:33:47.399193 ip-10-0-131-134 systemd[1]: Started Kubernetes Kubelet. Sep 16 07:33:47.400545 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.400318 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 07:33:47.400545 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.400361 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-134.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 07:33:47.400545 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.400364 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 07:33:47.400545 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.400515 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 07:33:47.402514 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.402356 2580 server.go:310] "Adding debug handlers to kubelet server" Sep 16 07:33:47.404776 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.404751 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 07:33:47.405714 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.405700 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 07:33:47.406163 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.405241 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-134.ec2.internal.18d5bccfbefeee15 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-134.ec2.internal,UID:ip-10-0-131-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-134.ec2.internal,},FirstTimestamp:2026-09-16 07:33:47.398270485 +0000 UTC m=+0.317886422,LastTimestamp:2026-09-16 07:33:47.398270485 +0000 UTC m=+0.317886422,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-134.ec2.internal,}" Sep 16 07:33:47.406281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.406261 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 07:33:47.406510 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.406493 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:47.406984 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.406969 2580 factory.go:55] Registering systemd factory Sep 16 07:33:47.406984 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.406985 2580 factory.go:223] Registration of the systemd container factory successfully Sep 16 07:33:47.407099 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407078 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 07:33:47.407099 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407098 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 07:33:47.407268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407077 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 07:33:47.407268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407201 2580 factory.go:153] Registering CRI-O factory Sep 16 07:33:47.407268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407222 2580 reconstruct.go:97] "Volume reconstruction finished" Sep 16 07:33:47.407268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407232 2580 reconciler.go:29] "Reconciler: start to sync state" Sep 16 07:33:47.407268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407232 2580 factory.go:223] Registration of the crio container factory successfully Sep 16 07:33:47.407476 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407289 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 07:33:47.407476 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407302 2580 factory.go:103] Registering Raw factory Sep 16 07:33:47.407476 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407313 2580 manager.go:1196] Started watching for new ooms in manager Sep 16 07:33:47.407681 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.407669 2580 manager.go:349] Starting recovery of all containers Sep 16 07:33:47.414836 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.414710 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 07:33:47.414836 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.414745 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 07:33:47.420698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.420683 2580 manager.go:354] Recovery completed Sep 16 07:33:47.426029 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.425936 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:47.427738 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.427720 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:47.427803 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.427749 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:47.427803 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.427759 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:47.428237 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.428222 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 07:33:47.428237 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.428235 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 07:33:47.428377 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.428240 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:47.428377 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.428252 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 16 07:33:47.428773 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.428751 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wd88h" Sep 16 07:33:47.428881 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.428868 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:47.428922 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.428895 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:47.428922 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.428909 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:47.429997 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.429922 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-134.ec2.internal.18d5bccfc0c08df1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-134.ec2.internal,UID:ip-10-0-131-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-134.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-134.ec2.internal,},FirstTimestamp:2026-09-16 07:33:47.427737073 +0000 UTC m=+0.347353013,LastTimestamp:2026-09-16 07:33:47.427737073 +0000 UTC m=+0.347353013,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-134.ec2.internal,}" Sep 16 07:33:47.430306 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.430294 2580 policy_none.go:49] "None policy: Start" Sep 16 07:33:47.430368 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.430310 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 07:33:47.430368 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.430321 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 07:33:47.432143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.432131 2580 policy_none.go:47] "Start" Sep 16 07:33:47.435404 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.435385 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wd88h" Sep 16 07:33:47.437045 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.436962 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-134.ec2.internal.18d5bccfc0c0cde7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-134.ec2.internal,UID:ip-10-0-131-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-134.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-134.ec2.internal,},FirstTimestamp:2026-09-16 07:33:47.427753447 +0000 UTC m=+0.347369384,LastTimestamp:2026-09-16 07:33:47.427753447 +0000 UTC m=+0.347369384,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-134.ec2.internal,}" Sep 16 07:33:47.483332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.467751 2580 manager.go:337] "Starting Device Plugin manager" Sep 16 07:33:47.483332 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.467786 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 07:33:47.483332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.467798 2580 server.go:87] "Starting device plugin registration server" Sep 16 07:33:47.483332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.468066 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 07:33:47.483332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.468078 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 07:33:47.483332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.468343 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 07:33:47.483332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.468395 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 07:33:47.483332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.468404 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 07:33:47.483332 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.468948 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 07:33:47.483332 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.468990 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:47.552303 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.552276 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 07:33:47.553387 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.553367 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 07:33:47.553472 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.553413 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 07:33:47.553472 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.553432 2580 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 07:33:47.553574 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.553519 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 07:33:47.555649 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.555628 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:47.568407 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.568387 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:47.571717 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.571703 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:47.571803 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.571729 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:47.571803 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.571740 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:47.571803 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.571761 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.579524 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.579458 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.579524 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.579482 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-134.ec2.internal\": node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:47.605755 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.605734 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:47.654203 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.654178 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal"] Sep 16 07:33:47.654275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.654266 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:47.655537 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.655524 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:47.655611 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.655550 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:47.655611 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.655564 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:47.657562 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.657550 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:47.657685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.657672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.657751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.657707 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:47.658191 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.658174 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:47.658283 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.658203 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:47.658283 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.658174 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:47.658283 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.658234 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:47.658283 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.658245 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:47.658283 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.658258 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:47.660159 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.660142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.660242 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.660174 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:47.660843 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.660828 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:47.660912 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.660853 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:47.660912 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.660865 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:47.674727 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.674709 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-134.ec2.internal\" not found" node="ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.677859 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.677842 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-134.ec2.internal\" not found" node="ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.706181 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.706163 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:47.708200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.708185 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41ed10fcde45f45450c28bb899cca9ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"41ed10fcde45f45450c28bb899cca9ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.708293 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.708227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3db43c1838ed8ed4417671dad4e9e6c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-134.ec2.internal\" (UID: \"f3db43c1838ed8ed4417671dad4e9e6c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.708293 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.708253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41ed10fcde45f45450c28bb899cca9ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"41ed10fcde45f45450c28bb899cca9ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.807265 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.807241 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:47.809356 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.809335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41ed10fcde45f45450c28bb899cca9ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"41ed10fcde45f45450c28bb899cca9ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.809459 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.809359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41ed10fcde45f45450c28bb899cca9ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"41ed10fcde45f45450c28bb899cca9ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.809459 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.809375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3db43c1838ed8ed4417671dad4e9e6c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-134.ec2.internal\" (UID: \"f3db43c1838ed8ed4417671dad4e9e6c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.809459 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.809400 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3db43c1838ed8ed4417671dad4e9e6c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-134.ec2.internal\" (UID: \"f3db43c1838ed8ed4417671dad4e9e6c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.809459 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.809433 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41ed10fcde45f45450c28bb899cca9ea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"41ed10fcde45f45450c28bb899cca9ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.809459 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.809438 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41ed10fcde45f45450c28bb899cca9ea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"41ed10fcde45f45450c28bb899cca9ea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.907832 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:47.907776 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:47.977076 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.977060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Sep 16 07:33:47.980250 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:47.980235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Sep 16 07:33:48.007933 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:48.007909 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:48.108351 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:48.108325 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:48.208796 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:48.208750 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:48.309148 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:48.309114 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:48.352319 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.352288 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 07:33:48.352768 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.352431 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 07:33:48.386672 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.386647 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:48.410126 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:48.410105 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:48.440704 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.440671 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 07:28:48 +0000 UTC" deadline="2028-02-28 07:21:26.813392714 +0000 UTC" Sep 16 07:33:48.440704 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.440700 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12719h47m38.372695276s" Sep 16 07:33:48.511163 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:48.511114 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:48.611565 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:48.611536 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:48.619786 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.619767 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:48.620983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.620966 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:48.621087 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.620997 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:48.621087 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.621008 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:48.711651 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:48.711626 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:48.716887 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:48.716855 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3db43c1838ed8ed4417671dad4e9e6c.slice/crio-a7f4946225de0adeb6a354a0ea67743bdad1839b0713170c7e444ac1ccc0e650 WatchSource:0}: Error finding container a7f4946225de0adeb6a354a0ea67743bdad1839b0713170c7e444ac1ccc0e650: Status 404 returned error can't find the container with id a7f4946225de0adeb6a354a0ea67743bdad1839b0713170c7e444ac1ccc0e650 Sep 16 07:33:48.721708 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.721688 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:33:48.812632 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:48.812564 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Sep 16 07:33:48.823632 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.823610 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:48.907342 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.907316 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Sep 16 07:33:48.924060 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.924034 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 07:33:48.926180 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.926164 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Sep 16 07:33:48.935976 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.935960 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 07:33:48.973555 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:48.973530 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ed10fcde45f45450c28bb899cca9ea.slice/crio-fdc892f6c9d49c97f8692a6d0909ab1da9b9705fa99a1ed42b3a5dc936bb1590 WatchSource:0}: Error finding container fdc892f6c9d49c97f8692a6d0909ab1da9b9705fa99a1ed42b3a5dc936bb1590: Status 404 returned error can't find the container with id fdc892f6c9d49c97f8692a6d0909ab1da9b9705fa99a1ed42b3a5dc936bb1590 Sep 16 07:33:48.986860 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:48.986838 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:49.388730 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.388694 2580 apiserver.go:52] "Watching apiserver" Sep 16 07:33:49.402143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.402116 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 07:33:49.402637 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.402610 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-zz4tf","openshift-multus/network-metrics-daemon-8qm8r","openshift-network-diagnostics/network-check-target-xsx7w","openshift-network-operator/iptables-alerter-h6lpf","kube-system/global-pull-secret-syncer-l2vjc","kube-system/konnectivity-agent-wscpg","kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq","openshift-cluster-node-tuning-operator/tuned-79m7m","openshift-dns/node-resolver-h69ht","openshift-multus/multus-additional-cni-plugins-6pl84","openshift-ovn-kubernetes/ovnkube-node-zncgj","openshift-image-registry/node-ca-9w78q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal"] Sep 16 07:33:49.405315 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.405298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.406312 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.406292 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 07:33:49.407642 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.407618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.410106 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.410085 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:49.410213 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.410170 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:33:49.412354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.412333 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.412482 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.412462 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 07:33:49.412551 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.412503 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 07:33:49.412955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.412932 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 07:33:49.413073 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.413055 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4tdj5\"" Sep 16 07:33:49.413928 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.413910 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 07:33:49.414166 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.414140 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2pvw7\"" Sep 16 07:33:49.414382 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.414364 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 07:33:49.414634 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.414611 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:49.414730 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.414688 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:33:49.417141 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.417122 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:33:49.419353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.419258 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 07:33:49.419353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.419310 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:33:49.419562 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.419544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.419852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.419829 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 07:33:49.420017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.419994 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bmsbq\"" Sep 16 07:33:49.420555 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.420533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhbs5\" (UniqueName: \"kubernetes.io/projected/5b2f9240-4d84-45e6-bfc3-fa31321bda0c-kube-api-access-bhbs5\") pod \"node-resolver-h69ht\" (UID: \"5b2f9240-4d84-45e6-bfc3-fa31321bda0c\") " pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.420649 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.420567 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b2f9240-4d84-45e6-bfc3-fa31321bda0c-hosts-file\") pod \"node-resolver-h69ht\" (UID: \"5b2f9240-4d84-45e6-bfc3-fa31321bda0c\") " pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.420649 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.420595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2f9240-4d84-45e6-bfc3-fa31321bda0c-tmp-dir\") pod \"node-resolver-h69ht\" (UID: \"5b2f9240-4d84-45e6-bfc3-fa31321bda0c\") " pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.420649 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.420619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36dbd8bf-bd64-4f3c-9d7f-ab618f797a96-host\") pod \"node-ca-9w78q\" (UID: \"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96\") " pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.420649 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.420645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36dbd8bf-bd64-4f3c-9d7f-ab618f797a96-serviceca\") pod \"node-ca-9w78q\" (UID: \"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96\") " pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.420874 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.420673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwqg9\" (UniqueName: \"kubernetes.io/projected/36dbd8bf-bd64-4f3c-9d7f-ab618f797a96-kube-api-access-qwqg9\") pod \"node-ca-9w78q\" (UID: \"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96\") " pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.422544 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.422429 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.422798 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.422781 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 07:33:49.423105 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.423089 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 07:33:49.423355 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.423339 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7k7p6\"" Sep 16 07:33:49.431698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.430951 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:33:49.431698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.431093 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 07:33:49.431698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.431112 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 07:33:49.431698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.431214 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lfzwn\"" Sep 16 07:33:49.431698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.431229 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 07:33:49.431698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.431328 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7gnkh\"" Sep 16 07:33:49.431698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.431448 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 07:33:49.431698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.431480 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:49.431698 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.431536 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:33:49.433934 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.433910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.436984 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.436381 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.439003 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.438980 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.441799 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.441758 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 07:33:49.441799 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.441789 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 07:33:49.442043 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442021 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 07:33:49.442131 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442045 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 07:33:49.442131 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442079 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 07:33:49.442249 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442036 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 07:33:49.442249 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442023 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dccdq\"" Sep 16 07:33:49.442397 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442305 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 07:33:49.442397 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442308 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xl8dw\"" Sep 16 07:33:49.442397 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442388 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 07:33:49.442576 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442550 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 07:33:49.442617 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442587 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 07:33:49.442920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.442899 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 07:33:49.443152 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.443135 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sgvqd\"" Sep 16 07:33:49.443466 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.443441 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 07:33:49.443660 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.443641 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 07:33:49.481722 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.481555 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fn9dc" Sep 16 07:33:49.488336 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.488308 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fn9dc" Sep 16 07:33:49.508356 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.508333 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 07:33:49.521745 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.521382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05675886-b0f7-47ae-b1ee-355badf36fff-cni-binary-copy\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.521849 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.521813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2270b6d5-4d64-43d1-8130-c30279ab526b-konnectivity-ca\") pod \"konnectivity-agent-wscpg\" (UID: \"2270b6d5-4d64-43d1-8130-c30279ab526b\") " pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:33:49.521916 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.521868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-run-netns\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.521974 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.521916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-kubelet\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.522024 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.521984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cee8f478-ddc3-463d-85fc-ac487e43e1b0-iptables-alerter-script\") pod \"iptables-alerter-h6lpf\" (UID: \"cee8f478-ddc3-463d-85fc-ac487e43e1b0\") " pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.522075 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36dbd8bf-bd64-4f3c-9d7f-ab618f797a96-host\") pod \"node-ca-9w78q\" (UID: \"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96\") " pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.522075 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-slash\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.522165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-run-systemd\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.522165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36dbd8bf-bd64-4f3c-9d7f-ab618f797a96-host\") pod \"node-ca-9w78q\" (UID: \"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96\") " pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.522165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-var-lib-openvswitch\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.522337 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-env-overrides\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.522406 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-ovn-node-metrics-cert\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.522459 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9tqq\" (UniqueName: \"kubernetes.io/projected/cee8f478-ddc3-463d-85fc-ac487e43e1b0-kube-api-access-z9tqq\") pod \"iptables-alerter-h6lpf\" (UID: \"cee8f478-ddc3-463d-85fc-ac487e43e1b0\") " pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.522573 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-modprobe-d\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.522573 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-run-netns\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.522573 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-run-openvswitch\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.522721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clg5c\" (UniqueName: \"kubernetes.io/projected/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-kube-api-access-clg5c\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.522721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-kubernetes\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.522721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-run\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.522721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-sys\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.522906 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-lib-modules\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.522906 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-host\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.522906 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2f9240-4d84-45e6-bfc3-fa31321bda0c-tmp-dir\") pod \"node-resolver-h69ht\" (UID: \"5b2f9240-4d84-45e6-bfc3-fa31321bda0c\") " pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.522906 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522837 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-run-multus-certs\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.523082 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-etc-kubernetes\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.523082 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-ovnkube-config\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.523082 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.522977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cee8f478-ddc3-463d-85fc-ac487e43e1b0-host-slash\") pod \"iptables-alerter-h6lpf\" (UID: \"cee8f478-ddc3-463d-85fc-ac487e43e1b0\") " pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.523082 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-registration-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.523082 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-sys-fs\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.523353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkjg\" (UniqueName: \"kubernetes.io/projected/ba298277-ba49-4a22-8b0b-27ff1b2b024e-kube-api-access-xvkjg\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.523353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-systemd-units\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.523353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-cnibin\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.523353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8065977b-91e5-4fed-8250-5d4f379c05e3-cni-binary-copy\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.523353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-hostroot\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.523353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-run-ovn\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.523353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b40f3f8d-04c4-4581-b929-fabed41b7079-kubelet-config\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:49.523353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523344 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2f9240-4d84-45e6-bfc3-fa31321bda0c-tmp-dir\") pod \"node-resolver-h69ht\" (UID: \"5b2f9240-4d84-45e6-bfc3-fa31321bda0c\") " pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.523721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-systemd\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.523721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05675886-b0f7-47ae-b1ee-355badf36fff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.523721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b2f9240-4d84-45e6-bfc3-fa31321bda0c-hosts-file\") pod \"node-resolver-h69ht\" (UID: \"5b2f9240-4d84-45e6-bfc3-fa31321bda0c\") " pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.523721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwqg9\" (UniqueName: \"kubernetes.io/projected/36dbd8bf-bd64-4f3c-9d7f-ab618f797a96-kube-api-access-qwqg9\") pod \"node-ca-9w78q\" (UID: \"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96\") " pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.523721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b2f9240-4d84-45e6-bfc3-fa31321bda0c-hosts-file\") pod \"node-resolver-h69ht\" (UID: \"5b2f9240-4d84-45e6-bfc3-fa31321bda0c\") " pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.523721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-cni-bin\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.523991 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.523991 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05675886-b0f7-47ae-b1ee-355badf36fff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.523991 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6v9\" (UniqueName: \"kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9\") pod \"network-check-target-xsx7w\" (UID: \"720450fe-3414-486b-b860-59fbdb3cfda5\") " pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:49.523991 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.523975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-socket-dir-parent\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.524275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-conf-dir\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.524275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-log-socket\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.524275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-ovnkube-script-lib\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.524275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524124 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:49.524275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-system-cni-dir\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.524275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-var-lib-cni-multus\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.524275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:49.524275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-sysctl-d\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk588\" (UniqueName: \"kubernetes.io/projected/523d8800-04c4-438e-98fe-c0e180fb387f-kube-api-access-nk588\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524379 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9bvl\" (UniqueName: \"kubernetes.io/projected/05675886-b0f7-47ae-b1ee-355badf36fff-kube-api-access-j9bvl\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-daemon-config\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-sysctl-conf\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/523d8800-04c4-438e-98fe-c0e180fb387f-etc-tuned\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/523d8800-04c4-438e-98fe-c0e180fb387f-tmp\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-var-lib-cni-bin\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-device-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmkk\" (UniqueName: \"kubernetes.io/projected/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-kube-api-access-zmmkk\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:49.524689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-cni-dir\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-etc-openvswitch\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05675886-b0f7-47ae-b1ee-355badf36fff-system-cni-dir\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b40f3f8d-04c4-4581-b929-fabed41b7079-dbus\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524906 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-sysconfig\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2270b6d5-4d64-43d1-8130-c30279ab526b-agent-certs\") pod \"konnectivity-agent-wscpg\" (UID: \"2270b6d5-4d64-43d1-8130-c30279ab526b\") " pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.524970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-var-lib-kubelet\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-cni-netd\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-socket-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhbs5\" (UniqueName: \"kubernetes.io/projected/5b2f9240-4d84-45e6-bfc3-fa31321bda0c-kube-api-access-bhbs5\") pod \"node-resolver-h69ht\" (UID: \"5b2f9240-4d84-45e6-bfc3-fa31321bda0c\") " pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.525244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-run-k8s-cni-cncf-io\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.525768 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-var-lib-kubelet\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.525768 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05675886-b0f7-47ae-b1ee-355badf36fff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.525768 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36dbd8bf-bd64-4f3c-9d7f-ab618f797a96-serviceca\") pod \"node-ca-9w78q\" (UID: \"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96\") " pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.525768 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcwkz\" (UniqueName: \"kubernetes.io/projected/8065977b-91e5-4fed-8250-5d4f379c05e3-kube-api-access-tcwkz\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.525768 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525426 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-node-log\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.525768 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.525500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05675886-b0f7-47ae-b1ee-355badf36fff-cnibin\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.526563 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.526065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36dbd8bf-bd64-4f3c-9d7f-ab618f797a96-serviceca\") pod \"node-ca-9w78q\" (UID: \"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96\") " pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.530542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.530515 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 07:33:49.536321 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.536280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwqg9\" (UniqueName: \"kubernetes.io/projected/36dbd8bf-bd64-4f3c-9d7f-ab618f797a96-kube-api-access-qwqg9\") pod \"node-ca-9w78q\" (UID: \"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96\") " pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.538455 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.538434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhbs5\" (UniqueName: \"kubernetes.io/projected/5b2f9240-4d84-45e6-bfc3-fa31321bda0c-kube-api-access-bhbs5\") pod \"node-resolver-h69ht\" (UID: \"5b2f9240-4d84-45e6-bfc3-fa31321bda0c\") " pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.559954 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.559910 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" event={"ID":"41ed10fcde45f45450c28bb899cca9ea","Type":"ContainerStarted","Data":"fdc892f6c9d49c97f8692a6d0909ab1da9b9705fa99a1ed42b3a5dc936bb1590"} Sep 16 07:33:49.561399 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.561372 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" event={"ID":"f3db43c1838ed8ed4417671dad4e9e6c","Type":"ContainerStarted","Data":"a7f4946225de0adeb6a354a0ea67743bdad1839b0713170c7e444ac1ccc0e650"} Sep 16 07:33:49.625749 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-cni-bin\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.625874 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.625874 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05675886-b0f7-47ae-b1ee-355badf36fff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.625874 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6v9\" (UniqueName: \"kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9\") pod \"network-check-target-xsx7w\" (UID: \"720450fe-3414-486b-b860-59fbdb3cfda5\") " pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:49.625874 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-socket-dir-parent\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.625874 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.625874 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-conf-dir\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.626143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-log-socket\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.626143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-conf-dir\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.626143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-cni-bin\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.626143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-socket-dir-parent\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.626143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.625995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-ovnkube-script-lib\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.626143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:49.626143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-system-cni-dir\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.626143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-var-lib-cni-multus\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.626143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-sysctl-d\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk588\" (UniqueName: \"kubernetes.io/projected/523d8800-04c4-438e-98fe-c0e180fb387f-kube-api-access-nk588\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9bvl\" (UniqueName: \"kubernetes.io/projected/05675886-b0f7-47ae-b1ee-355badf36fff-kube-api-access-j9bvl\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-daemon-config\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-sysctl-conf\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626312 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-system-cni-dir\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/523d8800-04c4-438e-98fe-c0e180fb387f-etc-tuned\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626000 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-log-socket\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/523d8800-04c4-438e-98fe-c0e180fb387f-tmp\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-var-lib-cni-bin\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.626438 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:49.626471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-ovnkube-script-lib\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-device-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.626499 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs podName:533591c0-bcd3-4f1f-9ba5-3eb2effa48a1 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:50.126479514 +0000 UTC m=+3.046095459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs") pod "network-metrics-daemon-8qm8r" (UID: "533591c0-bcd3-4f1f-9ba5-3eb2effa48a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626543 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-var-lib-cni-bin\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626546 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-var-lib-cni-multus\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626590 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.626616 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-sysctl-conf\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-device-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626768 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-sysctl-d\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.626913 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626496 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/05675886-b0f7-47ae-b1ee-355badf36fff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626938 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.626967 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret podName:b40f3f8d-04c4-4581-b929-fabed41b7079 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:50.126947155 +0000 UTC m=+3.046563091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret") pod "global-pull-secret-syncer-l2vjc" (UID: "b40f3f8d-04c4-4581-b929-fabed41b7079") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.626987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmkk\" (UniqueName: \"kubernetes.io/projected/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-kube-api-access-zmmkk\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-cni-dir\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-etc-openvswitch\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05675886-b0f7-47ae-b1ee-355badf36fff-system-cni-dir\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b40f3f8d-04c4-4581-b929-fabed41b7079-dbus\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-etc-openvswitch\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-cni-dir\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-sysconfig\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2270b6d5-4d64-43d1-8130-c30279ab526b-agent-certs\") pod \"konnectivity-agent-wscpg\" (UID: \"2270b6d5-4d64-43d1-8130-c30279ab526b\") " pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627174 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-sysconfig\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05675886-b0f7-47ae-b1ee-355badf36fff-system-cni-dir\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-var-lib-kubelet\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b40f3f8d-04c4-4581-b929-fabed41b7079-dbus\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-var-lib-kubelet\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.627492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8065977b-91e5-4fed-8250-5d4f379c05e3-multus-daemon-config\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627312 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-cni-netd\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-cni-netd\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-socket-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-run-k8s-cni-cncf-io\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-var-lib-kubelet\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627465 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-run-k8s-cni-cncf-io\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05675886-b0f7-47ae-b1ee-355badf36fff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627438 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627511 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-socket-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627577 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-var-lib-kubelet\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcwkz\" (UniqueName: \"kubernetes.io/projected/8065977b-91e5-4fed-8250-5d4f379c05e3-kube-api-access-tcwkz\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-node-log\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-node-log\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05675886-b0f7-47ae-b1ee-355badf36fff-cnibin\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05675886-b0f7-47ae-b1ee-355badf36fff-cni-binary-copy\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.628354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2270b6d5-4d64-43d1-8130-c30279ab526b-konnectivity-ca\") pod \"konnectivity-agent-wscpg\" (UID: \"2270b6d5-4d64-43d1-8130-c30279ab526b\") " pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-run-netns\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05675886-b0f7-47ae-b1ee-355badf36fff-cnibin\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-kubelet\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/05675886-b0f7-47ae-b1ee-355badf36fff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cee8f478-ddc3-463d-85fc-ac487e43e1b0-iptables-alerter-script\") pod \"iptables-alerter-h6lpf\" (UID: \"cee8f478-ddc3-463d-85fc-ac487e43e1b0\") " pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627929 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-kubelet\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-slash\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.627991 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-run-netns\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05675886-b0f7-47ae-b1ee-355badf36fff-cni-binary-copy\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-run-systemd\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-var-lib-openvswitch\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-env-overrides\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-ovn-node-metrics-cert\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tqq\" (UniqueName: \"kubernetes.io/projected/cee8f478-ddc3-463d-85fc-ac487e43e1b0-kube-api-access-z9tqq\") pod \"iptables-alerter-h6lpf\" (UID: \"cee8f478-ddc3-463d-85fc-ac487e43e1b0\") " pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-modprobe-d\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-run-netns\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.629174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-run-openvswitch\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clg5c\" (UniqueName: \"kubernetes.io/projected/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-kube-api-access-clg5c\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-kubernetes\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-run\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-sys\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-lib-modules\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628455 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-slash\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628429 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cee8f478-ddc3-463d-85fc-ac487e43e1b0-iptables-alerter-script\") pod \"iptables-alerter-h6lpf\" (UID: \"cee8f478-ddc3-463d-85fc-ac487e43e1b0\") " pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628496 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-host\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628502 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-run\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-run-multus-certs\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628545 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-host-run-multus-certs\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628582 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-host-run-netns\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2270b6d5-4d64-43d1-8130-c30279ab526b-konnectivity-ca\") pod \"konnectivity-agent-wscpg\" (UID: \"2270b6d5-4d64-43d1-8130-c30279ab526b\") " pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-lib-modules\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628747 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-kubernetes\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-modprobe-d\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-sys\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.630051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-etc-kubernetes\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-run-openvswitch\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-var-lib-openvswitch\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-host\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-run-systemd\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-ovnkube-config\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.628941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cee8f478-ddc3-463d-85fc-ac487e43e1b0-host-slash\") pod \"iptables-alerter-h6lpf\" (UID: \"cee8f478-ddc3-463d-85fc-ac487e43e1b0\") " pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-registration-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-sys-fs\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkjg\" (UniqueName: \"kubernetes.io/projected/ba298277-ba49-4a22-8b0b-27ff1b2b024e-kube-api-access-xvkjg\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-systemd-units\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-cnibin\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8065977b-91e5-4fed-8250-5d4f379c05e3-cni-binary-copy\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-env-overrides\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-hostroot\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-cnibin\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-run-ovn\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629500 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-etc-kubernetes\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.631017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-ovnkube-config\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b40f3f8d-04c4-4581-b929-fabed41b7079-kubelet-config\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629539 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cee8f478-ddc3-463d-85fc-ac487e43e1b0-host-slash\") pod \"iptables-alerter-h6lpf\" (UID: \"cee8f478-ddc3-463d-85fc-ac487e43e1b0\") " pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-systemd\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b40f3f8d-04c4-4581-b929-fabed41b7079-kubelet-config\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-run-ovn\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05675886-b0f7-47ae-b1ee-355badf36fff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629639 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-systemd-units\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/523d8800-04c4-438e-98fe-c0e180fb387f-etc-systemd\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-sys-fs\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629776 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ba298277-ba49-4a22-8b0b-27ff1b2b024e-registration-dir\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05675886-b0f7-47ae-b1ee-355badf36fff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629876 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8065977b-91e5-4fed-8250-5d4f379c05e3-hostroot\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.629975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8065977b-91e5-4fed-8250-5d4f379c05e3-cni-binary-copy\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.630162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/523d8800-04c4-438e-98fe-c0e180fb387f-etc-tuned\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.630200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/523d8800-04c4-438e-98fe-c0e180fb387f-tmp\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.631265 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2270b6d5-4d64-43d1-8130-c30279ab526b-agent-certs\") pod \"konnectivity-agent-wscpg\" (UID: \"2270b6d5-4d64-43d1-8130-c30279ab526b\") " pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:33:49.631751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.631473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-ovn-node-metrics-cert\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.634567 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.634383 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:33:49.634567 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.634405 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:33:49.634567 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.634416 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mj6v9 for pod openshift-network-diagnostics/network-check-target-xsx7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:49.634567 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:49.634481 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9 podName:720450fe-3414-486b-b860-59fbdb3cfda5 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:50.134465608 +0000 UTC m=+3.054081539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mj6v9" (UniqueName: "kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9") pod "network-check-target-xsx7w" (UID: "720450fe-3414-486b-b860-59fbdb3cfda5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:49.636576 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.636557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9bvl\" (UniqueName: \"kubernetes.io/projected/05675886-b0f7-47ae-b1ee-355badf36fff-kube-api-access-j9bvl\") pod \"multus-additional-cni-plugins-6pl84\" (UID: \"05675886-b0f7-47ae-b1ee-355badf36fff\") " pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.637030 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.637007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk588\" (UniqueName: \"kubernetes.io/projected/523d8800-04c4-438e-98fe-c0e180fb387f-kube-api-access-nk588\") pod \"tuned-79m7m\" (UID: \"523d8800-04c4-438e-98fe-c0e180fb387f\") " pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.637564 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.637540 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmkk\" (UniqueName: \"kubernetes.io/projected/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-kube-api-access-zmmkk\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:49.640035 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.639984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tqq\" (UniqueName: \"kubernetes.io/projected/cee8f478-ddc3-463d-85fc-ac487e43e1b0-kube-api-access-z9tqq\") pod \"iptables-alerter-h6lpf\" (UID: \"cee8f478-ddc3-463d-85fc-ac487e43e1b0\") " pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.640302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.640237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clg5c\" (UniqueName: \"kubernetes.io/projected/95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e-kube-api-access-clg5c\") pod \"ovnkube-node-zncgj\" (UID: \"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.640521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.640500 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcwkz\" (UniqueName: \"kubernetes.io/projected/8065977b-91e5-4fed-8250-5d4f379c05e3-kube-api-access-tcwkz\") pod \"multus-zz4tf\" (UID: \"8065977b-91e5-4fed-8250-5d4f379c05e3\") " pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.640914 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.640896 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkjg\" (UniqueName: \"kubernetes.io/projected/ba298277-ba49-4a22-8b0b-27ff1b2b024e-kube-api-access-xvkjg\") pod \"aws-ebs-csi-driver-node-2prrq\" (UID: \"ba298277-ba49-4a22-8b0b-27ff1b2b024e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.717572 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.717544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h69ht" Sep 16 07:33:49.730514 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.730493 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9w78q" Sep 16 07:33:49.730690 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:49.730664 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2f9240_4d84_45e6_bfc3_fa31321bda0c.slice/crio-5dde1d09592680cab7079968d4924e33050e907f766ce15301ff772ae85c6565 WatchSource:0}: Error finding container 5dde1d09592680cab7079968d4924e33050e907f766ce15301ff772ae85c6565: Status 404 returned error can't find the container with id 5dde1d09592680cab7079968d4924e33050e907f766ce15301ff772ae85c6565 Sep 16 07:33:49.741050 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:49.740988 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dbd8bf_bd64_4f3c_9d7f_ab618f797a96.slice/crio-b2f0e1c69159131c935902b790ef66010721c01c0eff771f1de121fca9491a84 WatchSource:0}: Error finding container b2f0e1c69159131c935902b790ef66010721c01c0eff771f1de121fca9491a84: Status 404 returned error can't find the container with id b2f0e1c69159131c935902b790ef66010721c01c0eff771f1de121fca9491a84 Sep 16 07:33:49.743999 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.743980 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h6lpf" Sep 16 07:33:49.751466 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.751445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:33:49.753367 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:49.753346 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee8f478_ddc3_463d_85fc_ac487e43e1b0.slice/crio-edd483c2242c114f3cac656378693b2af444c62bd59cb071a8bc4d1fd04acaf1 WatchSource:0}: Error finding container edd483c2242c114f3cac656378693b2af444c62bd59cb071a8bc4d1fd04acaf1: Status 404 returned error can't find the container with id edd483c2242c114f3cac656378693b2af444c62bd59cb071a8bc4d1fd04acaf1 Sep 16 07:33:49.759967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.759943 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-79m7m" Sep 16 07:33:49.768806 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.768786 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" Sep 16 07:33:49.776130 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.775841 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zz4tf" Sep 16 07:33:49.785784 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.785765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6pl84" Sep 16 07:33:49.791846 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.791828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:33:49.888501 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:49.888475 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:50.134246 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:50.134151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:50.134246 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:50.134198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:50.134473 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:50.134313 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:50.134473 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:50.134326 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:50.134473 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:50.134393 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs podName:533591c0-bcd3-4f1f-9ba5-3eb2effa48a1 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:51.134371324 +0000 UTC m=+4.053987272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs") pod "network-metrics-daemon-8qm8r" (UID: "533591c0-bcd3-4f1f-9ba5-3eb2effa48a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:50.134473 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:50.134415 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret podName:b40f3f8d-04c4-4581-b929-fabed41b7079 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:51.134405518 +0000 UTC m=+4.054021444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret") pod "global-pull-secret-syncer-l2vjc" (UID: "b40f3f8d-04c4-4581-b929-fabed41b7079") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:50.234789 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:50.234752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6v9\" (UniqueName: \"kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9\") pod \"network-check-target-xsx7w\" (UID: \"720450fe-3414-486b-b860-59fbdb3cfda5\") " pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:50.234957 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:50.234919 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:33:50.234957 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:50.234937 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:33:50.234957 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:50.234948 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mj6v9 for pod openshift-network-diagnostics/network-check-target-xsx7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:50.235362 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:50.235330 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9 podName:720450fe-3414-486b-b860-59fbdb3cfda5 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:51.234986069 +0000 UTC m=+4.154601996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj6v9" (UniqueName: "kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9") pod "network-check-target-xsx7w" (UID: "720450fe-3414-486b-b860-59fbdb3cfda5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:50.492460 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:50.492338 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 07:28:50 +0000 UTC" deadline="2028-04-12 14:09:11.283827224 +0000 UTC" Sep 16 07:33:50.492460 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:50.492412 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13782h35m20.791419952s" Sep 16 07:33:50.564495 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:50.564453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h6lpf" event={"ID":"cee8f478-ddc3-463d-85fc-ac487e43e1b0","Type":"ContainerStarted","Data":"edd483c2242c114f3cac656378693b2af444c62bd59cb071a8bc4d1fd04acaf1"} Sep 16 07:33:50.565588 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:50.565559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9w78q" event={"ID":"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96","Type":"ContainerStarted","Data":"b2f0e1c69159131c935902b790ef66010721c01c0eff771f1de121fca9491a84"} Sep 16 07:33:50.566675 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:50.566648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h69ht" event={"ID":"5b2f9240-4d84-45e6-bfc3-fa31321bda0c","Type":"ContainerStarted","Data":"5dde1d09592680cab7079968d4924e33050e907f766ce15301ff772ae85c6565"} Sep 16 07:33:51.103600 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:51.103563 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523d8800_04c4_438e_98fe_c0e180fb387f.slice/crio-4269983e7ed0545f965b3f8a82d69a2e9b2ee5e2577bfb6fc06216af83b5c3d5 WatchSource:0}: Error finding container 4269983e7ed0545f965b3f8a82d69a2e9b2ee5e2577bfb6fc06216af83b5c3d5: Status 404 returned error can't find the container with id 4269983e7ed0545f965b3f8a82d69a2e9b2ee5e2577bfb6fc06216af83b5c3d5 Sep 16 07:33:51.131810 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:33:51.131544 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05675886_b0f7_47ae_b1ee_355badf36fff.slice/crio-2adf99b44f19c73e368ac1e874280790ffc24e2a2e195cdc6e415da2f5386c38 WatchSource:0}: Error finding container 2adf99b44f19c73e368ac1e874280790ffc24e2a2e195cdc6e415da2f5386c38: Status 404 returned error can't find the container with id 2adf99b44f19c73e368ac1e874280790ffc24e2a2e195cdc6e415da2f5386c38 Sep 16 07:33:51.142588 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.142559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:51.142647 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.142608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:51.142748 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.142732 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:51.142810 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.142791 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret podName:b40f3f8d-04c4-4581-b929-fabed41b7079 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:53.142772369 +0000 UTC m=+6.062388300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret") pod "global-pull-secret-syncer-l2vjc" (UID: "b40f3f8d-04c4-4581-b929-fabed41b7079") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:51.142887 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.142859 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:51.142944 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.142894 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs podName:533591c0-bcd3-4f1f-9ba5-3eb2effa48a1 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:53.142881996 +0000 UTC m=+6.062497928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs") pod "network-metrics-daemon-8qm8r" (UID: "533591c0-bcd3-4f1f-9ba5-3eb2effa48a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:51.243615 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.243577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6v9\" (UniqueName: \"kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9\") pod \"network-check-target-xsx7w\" (UID: \"720450fe-3414-486b-b860-59fbdb3cfda5\") " pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:51.243802 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.243781 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:33:51.243871 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.243806 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:33:51.243871 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.243819 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mj6v9 for pod openshift-network-diagnostics/network-check-target-xsx7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:51.243978 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.243878 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9 podName:720450fe-3414-486b-b860-59fbdb3cfda5 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:53.243859975 +0000 UTC m=+6.163475916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj6v9" (UniqueName: "kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9") pod "network-check-target-xsx7w" (UID: "720450fe-3414-486b-b860-59fbdb3cfda5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:51.492999 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.492958 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 07:28:50 +0000 UTC" deadline="2028-06-11 07:37:17.898549698 +0000 UTC" Sep 16 07:33:51.492999 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.492990 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15216h3m26.405564515s" Sep 16 07:33:51.553775 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.553748 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:51.553914 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.553886 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:33:51.554502 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.554343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:51.554502 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.554468 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:33:51.554502 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.554490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:51.554720 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:51.554600 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:33:51.571499 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.571468 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" event={"ID":"ba298277-ba49-4a22-8b0b-27ff1b2b024e","Type":"ContainerStarted","Data":"0495903aef2d29045adf4824afdb5d4ec234756c51a12ad52ba600c7227f5b21"} Sep 16 07:33:51.573199 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.573174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pl84" event={"ID":"05675886-b0f7-47ae-b1ee-355badf36fff","Type":"ContainerStarted","Data":"2adf99b44f19c73e368ac1e874280790ffc24e2a2e195cdc6e415da2f5386c38"} Sep 16 07:33:51.574984 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.574882 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" event={"ID":"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e","Type":"ContainerStarted","Data":"948d4ee4e69cbb6b893cd744bbb57852b64a2cb25654a492ee13b27447537161"} Sep 16 07:33:51.576637 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.576550 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zz4tf" event={"ID":"8065977b-91e5-4fed-8250-5d4f379c05e3","Type":"ContainerStarted","Data":"023f34f574c0bbb2eee0e9e4c070a63ecca655355211ec11e38a5a8a435316b8"} Sep 16 07:33:51.578819 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.578764 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-79m7m" event={"ID":"523d8800-04c4-438e-98fe-c0e180fb387f","Type":"ContainerStarted","Data":"4269983e7ed0545f965b3f8a82d69a2e9b2ee5e2577bfb6fc06216af83b5c3d5"} Sep 16 07:33:51.585196 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.585152 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" event={"ID":"41ed10fcde45f45450c28bb899cca9ea","Type":"ContainerStarted","Data":"5ad1d3ad69c8740660598069c7d77d8db3dcc6c7d18595bc09046dbb02e68e55"} Sep 16 07:33:51.588999 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.588969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" event={"ID":"f3db43c1838ed8ed4417671dad4e9e6c","Type":"ContainerStarted","Data":"ed5cc07e6de5131423bcfca26b7b7a00404d3cc37b2b2069a4965481e3d2b58d"} Sep 16 07:33:51.591244 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:51.591181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wscpg" event={"ID":"2270b6d5-4d64-43d1-8130-c30279ab526b","Type":"ContainerStarted","Data":"0e56f8fff530f20b562586af25858bcf2e87f0de24706b4703b36597649f5c18"} Sep 16 07:33:52.603974 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:52.603938 2580 generic.go:356] "Generic (PLEG): container finished" podID="41ed10fcde45f45450c28bb899cca9ea" containerID="5ad1d3ad69c8740660598069c7d77d8db3dcc6c7d18595bc09046dbb02e68e55" exitCode=0 Sep 16 07:33:52.605085 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:52.605059 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" event={"ID":"41ed10fcde45f45450c28bb899cca9ea","Type":"ContainerDied","Data":"5ad1d3ad69c8740660598069c7d77d8db3dcc6c7d18595bc09046dbb02e68e55"} Sep 16 07:33:52.605180 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:52.605154 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" event={"ID":"41ed10fcde45f45450c28bb899cca9ea","Type":"ContainerStarted","Data":"3ddde7c74f1425b5591b72898f7f144a873fec26bdfe18232d462f4ddb8ad784"} Sep 16 07:33:52.622203 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:52.621100 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" podStartSLOduration=3.621080381 podStartE2EDuration="3.621080381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:33:51.627348245 +0000 UTC m=+4.546964196" watchObservedRunningTime="2026-09-16 07:33:52.621080381 +0000 UTC m=+5.540696331" Sep 16 07:33:53.161431 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:53.160679 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:53.161431 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:53.160734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:53.161431 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.160871 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:53.161431 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.160936 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret podName:b40f3f8d-04c4-4581-b929-fabed41b7079 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:57.160915943 +0000 UTC m=+10.080531871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret") pod "global-pull-secret-syncer-l2vjc" (UID: "b40f3f8d-04c4-4581-b929-fabed41b7079") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:53.161431 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.161002 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:53.161431 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.161037 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs podName:533591c0-bcd3-4f1f-9ba5-3eb2effa48a1 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:57.161024254 +0000 UTC m=+10.080640181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs") pod "network-metrics-daemon-8qm8r" (UID: "533591c0-bcd3-4f1f-9ba5-3eb2effa48a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:53.261955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:53.261917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6v9\" (UniqueName: \"kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9\") pod \"network-check-target-xsx7w\" (UID: \"720450fe-3414-486b-b860-59fbdb3cfda5\") " pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:53.262169 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.262138 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:33:53.262169 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.262165 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:33:53.262309 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.262180 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mj6v9 for pod openshift-network-diagnostics/network-check-target-xsx7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:53.262309 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.262241 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9 podName:720450fe-3414-486b-b860-59fbdb3cfda5 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:57.2622219 +0000 UTC m=+10.181837830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj6v9" (UniqueName: "kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9") pod "network-check-target-xsx7w" (UID: "720450fe-3414-486b-b860-59fbdb3cfda5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:53.559810 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:53.558879 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:53.559810 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.559034 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:33:53.559810 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:53.559467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:53.559810 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.559577 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:33:53.559810 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:53.559637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:53.559810 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:53.559721 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:33:55.555082 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:55.554561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:55.555082 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:55.554705 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:33:55.555082 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:55.554561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:55.555082 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:55.554849 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:33:55.555082 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:55.554902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:55.555082 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:55.554986 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:33:57.193114 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:57.193060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:57.193114 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:57.193115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:57.193656 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.193260 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:57.193656 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.193325 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret podName:b40f3f8d-04c4-4581-b929-fabed41b7079 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:05.193306364 +0000 UTC m=+18.112922306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret") pod "global-pull-secret-syncer-l2vjc" (UID: "b40f3f8d-04c4-4581-b929-fabed41b7079") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:57.193656 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.193372 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:57.193656 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.193444 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs podName:533591c0-bcd3-4f1f-9ba5-3eb2effa48a1 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:05.193423438 +0000 UTC m=+18.113039385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs") pod "network-metrics-daemon-8qm8r" (UID: "533591c0-bcd3-4f1f-9ba5-3eb2effa48a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:57.293652 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:57.293615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6v9\" (UniqueName: \"kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9\") pod \"network-check-target-xsx7w\" (UID: \"720450fe-3414-486b-b860-59fbdb3cfda5\") " pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:57.293837 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.293797 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:33:57.293837 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.293817 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:33:57.293837 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.293830 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mj6v9 for pod openshift-network-diagnostics/network-check-target-xsx7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:57.294023 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.293882 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9 podName:720450fe-3414-486b-b860-59fbdb3cfda5 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:05.293869342 +0000 UTC m=+18.213485272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj6v9" (UniqueName: "kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9") pod "network-check-target-xsx7w" (UID: "720450fe-3414-486b-b860-59fbdb3cfda5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:57.555387 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:57.555313 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:57.555387 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:57.555359 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:57.555598 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.555428 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:33:57.555598 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:57.555457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:57.555598 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.555556 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:33:57.555717 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:57.555630 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:33:59.554659 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:59.554615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:33:59.554659 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:59.554627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:33:59.555149 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:33:59.554750 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:33:59.555149 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:59.554785 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:33:59.555149 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:59.554812 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:33:59.555149 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:33:59.554925 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:01.554119 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:01.554044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:01.554119 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:01.554065 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:01.554119 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:01.554065 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:01.554675 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:01.554151 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:01.554675 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:01.554266 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:01.554675 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:01.554368 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:03.554447 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:03.554418 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:03.554447 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:03.554431 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:03.554949 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:03.554543 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:03.554949 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:03.554698 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:03.554949 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:03.554754 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:03.554949 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:03.554845 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:05.254700 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:05.254485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:05.255224 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:05.254712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:05.255224 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.254647 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:34:05.255224 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.254836 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs podName:533591c0-bcd3-4f1f-9ba5-3eb2effa48a1 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:21.254808955 +0000 UTC m=+34.174424892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs") pod "network-metrics-daemon-8qm8r" (UID: "533591c0-bcd3-4f1f-9ba5-3eb2effa48a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:34:05.255224 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.254838 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:34:05.255224 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.254879 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret podName:b40f3f8d-04c4-4581-b929-fabed41b7079 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:21.254868329 +0000 UTC m=+34.174484253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret") pod "global-pull-secret-syncer-l2vjc" (UID: "b40f3f8d-04c4-4581-b929-fabed41b7079") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:34:05.355536 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:05.355502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6v9\" (UniqueName: \"kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9\") pod \"network-check-target-xsx7w\" (UID: \"720450fe-3414-486b-b860-59fbdb3cfda5\") " pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:05.355697 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.355654 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:34:05.355697 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.355673 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:34:05.355697 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.355685 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mj6v9 for pod openshift-network-diagnostics/network-check-target-xsx7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:34:05.355863 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.355740 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9 podName:720450fe-3414-486b-b860-59fbdb3cfda5 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:21.355723331 +0000 UTC m=+34.275339262 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj6v9" (UniqueName: "kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9") pod "network-check-target-xsx7w" (UID: "720450fe-3414-486b-b860-59fbdb3cfda5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:34:05.553956 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:05.553879 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:05.553956 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:05.553895 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:05.553956 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:05.553926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:05.554164 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.554010 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:05.554164 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.554119 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:05.554263 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:05.554165 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:07.555092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:07.555061 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:07.555599 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:07.555168 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:07.555599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:07.555195 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:07.555599 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:07.555325 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:07.555599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:07.555370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:07.555599 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:07.555439 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:08.639492 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.637586 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9w78q" event={"ID":"36dbd8bf-bd64-4f3c-9d7f-ab618f797a96","Type":"ContainerStarted","Data":"980de794c7cc0920b6645ff2c8101a3f4ff5372acb7efb352ca1887017986976"} Sep 16 07:34:08.642188 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.642146 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h69ht" event={"ID":"5b2f9240-4d84-45e6-bfc3-fa31321bda0c","Type":"ContainerStarted","Data":"f8c5970a91e3bcb422cb9cb299ec71eb25490cb43c3e04eb1f48d04562aa940a"} Sep 16 07:34:08.643881 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.643853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wscpg" event={"ID":"2270b6d5-4d64-43d1-8130-c30279ab526b","Type":"ContainerStarted","Data":"ed8029b47fd06abb1476548ae4155b6a91a940c07bf307b6167ba656c785fc4d"} Sep 16 07:34:08.645981 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.645958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" event={"ID":"ba298277-ba49-4a22-8b0b-27ff1b2b024e","Type":"ContainerStarted","Data":"97d062f925f4efe065f2fb747ef31859e60a3b05a560b38f545ab582303c4303"} Sep 16 07:34:08.651130 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.651078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pl84" event={"ID":"05675886-b0f7-47ae-b1ee-355badf36fff","Type":"ContainerStarted","Data":"7f784b6b57afa8e3d553b661bf378667b551cbd7e6a55e5975aef241657e0a41"} Sep 16 07:34:08.652751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.652713 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9w78q" podStartSLOduration=2.032407397 podStartE2EDuration="20.652700055s" totalImagesPullingTime="18.620292658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:08.652135958 +0000 UTC m=+21.571751904" watchObservedRunningTime="2026-09-16 07:34:08.652700055 +0000 UTC m=+21.572316001" Sep 16 07:34:08.652883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.652786 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" podStartSLOduration=19.65278011 podStartE2EDuration="19.65278011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:33:52.621905798 +0000 UTC m=+5.541521748" watchObservedRunningTime="2026-09-16 07:34:08.65278011 +0000 UTC m=+21.572396057" Sep 16 07:34:08.653634 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.653613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" event={"ID":"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e","Type":"ContainerStarted","Data":"3cf231651a8bfcd3792950734519133161bf42350a989a1edf08e2baaba4c977"} Sep 16 07:34:08.655993 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.655809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zz4tf" event={"ID":"8065977b-91e5-4fed-8250-5d4f379c05e3","Type":"ContainerStarted","Data":"03f5ccdcc54566e578f51fff182f2c6e2311a0394b68f8e6e400561b3c48f46a"} Sep 16 07:34:08.658976 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.658943 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-79m7m" event={"ID":"523d8800-04c4-438e-98fe-c0e180fb387f","Type":"ContainerStarted","Data":"2dbb2b3f18972af57d2c3707a0469df62549556cbe865f8c64970a7c7ee20464"} Sep 16 07:34:08.692203 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.692162 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h69ht" podStartSLOduration=2.06185093 podStartE2EDuration="20.692147036s" totalImagesPullingTime="18.630296106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:08.691829853 +0000 UTC m=+21.611445800" watchObservedRunningTime="2026-09-16 07:34:08.692147036 +0000 UTC m=+21.611762983" Sep 16 07:34:08.696954 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.696928 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:34:08.697545 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.697521 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:34:08.706476 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.706432 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wscpg" podStartSLOduration=3.448318344 podStartE2EDuration="20.706420768s" totalImagesPullingTime="17.258102424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:08.706379797 +0000 UTC m=+21.625995744" watchObservedRunningTime="2026-09-16 07:34:08.706420768 +0000 UTC m=+21.626036714" Sep 16 07:34:08.724048 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.724007 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zz4tf" podStartSLOduration=3.414962503 podStartE2EDuration="20.723994364s" totalImagesPullingTime="17.309031861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:08.723730936 +0000 UTC m=+21.643346881" watchObservedRunningTime="2026-09-16 07:34:08.723994364 +0000 UTC m=+21.643610310" Sep 16 07:34:08.759169 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:08.759128 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-79m7m" podStartSLOduration=3.461163712 podStartE2EDuration="20.759115626s" totalImagesPullingTime="17.297951914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:08.758849426 +0000 UTC m=+21.678465366" watchObservedRunningTime="2026-09-16 07:34:08.759115626 +0000 UTC m=+21.678731571" Sep 16 07:34:09.554173 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.554144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:09.554173 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.554157 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:09.554503 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.554164 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:09.554503 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:09.554273 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:09.554503 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:09.554342 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:09.554503 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:09.554420 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:09.661800 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.661763 2580 generic.go:356] "Generic (PLEG): container finished" podID="05675886-b0f7-47ae-b1ee-355badf36fff" containerID="7f784b6b57afa8e3d553b661bf378667b551cbd7e6a55e5975aef241657e0a41" exitCode=0 Sep 16 07:34:09.662533 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.661845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pl84" event={"ID":"05675886-b0f7-47ae-b1ee-355badf36fff","Type":"ContainerDied","Data":"7f784b6b57afa8e3d553b661bf378667b551cbd7e6a55e5975aef241657e0a41"} Sep 16 07:34:09.665222 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.665183 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" event={"ID":"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e","Type":"ContainerStarted","Data":"335b7429b141bf2455f3228629e10301a6e6a5d398af142b6869d8514c7e1a20"} Sep 16 07:34:09.665305 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.665273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" event={"ID":"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e","Type":"ContainerStarted","Data":"a8d9067b6979085b355203037ce7c1811c301eb710192f735078285863b96a62"} Sep 16 07:34:09.665354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.665318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" event={"ID":"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e","Type":"ContainerStarted","Data":"65b180aaaa7366f031bc4628c22ae89ee096d75fcb4e42d26467e0e91bcfce72"} Sep 16 07:34:09.665397 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.665355 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" event={"ID":"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e","Type":"ContainerStarted","Data":"1f754a4c7d0f8f2391814346c6447efbecc0585a6cfb3140556c1528da992199"} Sep 16 07:34:09.665397 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.665393 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" event={"ID":"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e","Type":"ContainerStarted","Data":"4cf5054f351d27a6103a569d41738a52e5ca14f2161e49921ce0e69652702026"} Sep 16 07:34:09.665904 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.665887 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:34:09.665987 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.665944 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wscpg" Sep 16 07:34:09.884740 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:09.884607 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 07:34:10.482228 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:10.482089 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T07:34:09.88473456Z","UUID":"76766f5e-0e5b-49e7-b2b8-4ca07989fb55","Handler":null,"Name":"","Endpoint":""} Sep 16 07:34:10.483737 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:10.483717 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 07:34:10.483877 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:10.483758 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 07:34:10.670265 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:10.670228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h6lpf" event={"ID":"cee8f478-ddc3-463d-85fc-ac487e43e1b0","Type":"ContainerStarted","Data":"a927c2611193cdacb737360866a32d49762b72250c83681d70960e255fabf6e9"} Sep 16 07:34:10.673677 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:10.673652 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" event={"ID":"ba298277-ba49-4a22-8b0b-27ff1b2b024e","Type":"ContainerStarted","Data":"ab2827069d1a00e9331c299949ed377733d92764ae03a287177df441cd012080"} Sep 16 07:34:10.687314 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:10.687263 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h6lpf" podStartSLOduration=4.052074097 podStartE2EDuration="22.687248632s" totalImagesPullingTime="18.635174535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:10.68702943 +0000 UTC m=+23.606645377" watchObservedRunningTime="2026-09-16 07:34:10.687248632 +0000 UTC m=+23.606864568" Sep 16 07:34:11.553918 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:11.553889 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:11.553918 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:11.553902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:11.554284 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:11.554011 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:11.554284 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:11.554096 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:11.554284 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:11.554182 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:11.554284 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:11.554274 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:11.677879 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:11.677844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" event={"ID":"ba298277-ba49-4a22-8b0b-27ff1b2b024e","Type":"ContainerStarted","Data":"ebe39ec2c1d071215061e575bab1123ae41ff3f5e102cfaa7984dd3524ad1288"} Sep 16 07:34:11.681945 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:11.681576 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" event={"ID":"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e","Type":"ContainerStarted","Data":"944c68b84c831508aa04df81d3ecbd98680487a17009e52a49ced8808c1c286b"} Sep 16 07:34:11.697100 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:11.697063 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2prrq" podStartSLOduration=4.275794128 podStartE2EDuration="23.69705021s" totalImagesPullingTime="19.421256082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:11.696748265 +0000 UTC m=+24.616364212" watchObservedRunningTime="2026-09-16 07:34:11.69705021 +0000 UTC m=+24.616666156" Sep 16 07:34:13.553959 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:13.553880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:13.554362 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:13.554010 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:13.554362 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:13.553879 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:13.554362 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:13.554094 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:13.554362 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:13.553880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:13.554362 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:13.554176 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:14.691810 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:14.691616 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" event={"ID":"95f4cd94-a35a-4624-b1ff-8b3e1cf29f7e","Type":"ContainerStarted","Data":"6a690cb2e448f7068d40e2b1c79b55ee72dd650341fe570aad38c2e76c7905d5"} Sep 16 07:34:14.692100 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:14.691878 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:34:14.710545 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:14.710522 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:34:14.722781 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:14.722737 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" podStartSLOduration=9.405463195 podStartE2EDuration="26.722718785s" totalImagesPullingTime="17.31725559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:14.72229909 +0000 UTC m=+27.641915037" watchObservedRunningTime="2026-09-16 07:34:14.722718785 +0000 UTC m=+27.642334732" Sep 16 07:34:15.553820 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:15.553775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:15.554063 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:15.553783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:15.554063 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:15.553880 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:15.554063 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:15.553897 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:15.554063 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:15.553974 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:15.554228 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:15.554062 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:15.699548 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:15.697515 2580 generic.go:356] "Generic (PLEG): container finished" podID="05675886-b0f7-47ae-b1ee-355badf36fff" containerID="e3639afa0e845fb8618a79c1bf38fb99568a9d5c979c31586c93014cc51ceff4" exitCode=0 Sep 16 07:34:15.699548 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:15.699135 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pl84" event={"ID":"05675886-b0f7-47ae-b1ee-355badf36fff","Type":"ContainerDied","Data":"e3639afa0e845fb8618a79c1bf38fb99568a9d5c979c31586c93014cc51ceff4"} Sep 16 07:34:15.699548 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:15.699184 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:34:15.699548 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:15.699320 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:34:15.717204 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:15.717182 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:34:16.294728 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:16.294661 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l2vjc"] Sep 16 07:34:16.294869 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:16.294765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:16.294869 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:16.294854 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:16.297730 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:16.297704 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsx7w"] Sep 16 07:34:16.297861 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:16.297815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:16.297957 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:16.297935 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:16.298401 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:16.298376 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8qm8r"] Sep 16 07:34:16.298523 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:16.298475 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:16.298574 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:16.298555 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:17.555200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:17.555129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:17.555760 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:17.555225 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:17.555760 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:17.555263 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:17.555760 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:17.555308 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:17.704271 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:17.704238 2580 generic.go:356] "Generic (PLEG): container finished" podID="05675886-b0f7-47ae-b1ee-355badf36fff" containerID="c31e07b2d320acdc149e40a345b390df76db26d2a3fcc5e7e08fae0f435328e6" exitCode=0 Sep 16 07:34:17.704399 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:17.704318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pl84" event={"ID":"05675886-b0f7-47ae-b1ee-355badf36fff","Type":"ContainerDied","Data":"c31e07b2d320acdc149e40a345b390df76db26d2a3fcc5e7e08fae0f435328e6"} Sep 16 07:34:18.553969 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:18.553768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:18.554083 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:18.554002 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:18.709323 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:18.709295 2580 generic.go:356] "Generic (PLEG): container finished" podID="05675886-b0f7-47ae-b1ee-355badf36fff" containerID="eebda56a62dddb0ef1efed4102db3a250c8aa36ea4c28259a66ee72251695f51" exitCode=0 Sep 16 07:34:18.709641 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:18.709341 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pl84" event={"ID":"05675886-b0f7-47ae-b1ee-355badf36fff","Type":"ContainerDied","Data":"eebda56a62dddb0ef1efed4102db3a250c8aa36ea4c28259a66ee72251695f51"} Sep 16 07:34:19.554561 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:19.554527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:19.554765 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:19.554654 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:19.554915 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:19.554886 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:19.555024 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:19.555003 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:20.554530 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:20.554499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:20.554949 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:20.554609 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8qm8r" podUID="533591c0-bcd3-4f1f-9ba5-3eb2effa48a1" Sep 16 07:34:21.273843 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:21.273745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:21.273843 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:21.273794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:21.274118 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:21.273916 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:34:21.274118 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:21.273923 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:34:21.274118 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:21.273984 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret podName:b40f3f8d-04c4-4581-b929-fabed41b7079 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:53.273965519 +0000 UTC m=+66.193581460 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret") pod "global-pull-secret-syncer-l2vjc" (UID: "b40f3f8d-04c4-4581-b929-fabed41b7079") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:34:21.274118 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:21.274004 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs podName:533591c0-bcd3-4f1f-9ba5-3eb2effa48a1 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:53.273994535 +0000 UTC m=+66.193610465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs") pod "network-metrics-daemon-8qm8r" (UID: "533591c0-bcd3-4f1f-9ba5-3eb2effa48a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:34:21.375081 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:21.375052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6v9\" (UniqueName: \"kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9\") pod \"network-check-target-xsx7w\" (UID: \"720450fe-3414-486b-b860-59fbdb3cfda5\") " pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:21.375261 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:21.375172 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:34:21.375261 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:21.375193 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:34:21.375261 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:21.375223 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mj6v9 for pod openshift-network-diagnostics/network-check-target-xsx7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:34:21.375429 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:21.375283 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9 podName:720450fe-3414-486b-b860-59fbdb3cfda5 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:53.375265637 +0000 UTC m=+66.294881563 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mj6v9" (UniqueName: "kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9") pod "network-check-target-xsx7w" (UID: "720450fe-3414-486b-b860-59fbdb3cfda5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:34:21.554004 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:21.553930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:21.554155 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:21.553940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:21.554155 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:21.554048 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2vjc" podUID="b40f3f8d-04c4-4581-b929-fabed41b7079" Sep 16 07:34:21.554155 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:21.554116 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsx7w" podUID="720450fe-3414-486b-b860-59fbdb3cfda5" Sep 16 07:34:22.309721 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.309646 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeReady" Sep 16 07:34:22.310085 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.309773 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 07:34:22.348539 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.348511 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns"] Sep 16 07:34:22.391670 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.391637 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79ffc6588-khnvp"] Sep 16 07:34:22.391845 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.391787 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns" Sep 16 07:34:22.396055 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.395073 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9hzgv\"" Sep 16 07:34:22.396055 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.395408 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.396055 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.395721 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.405050 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.405027 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9"] Sep 16 07:34:22.405200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.405178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.408616 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.408591 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 07:34:22.409481 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.409458 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 07:34:22.409737 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.409722 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6bpwz\"" Sep 16 07:34:22.413088 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.410912 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 07:34:22.422092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.422074 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 07:34:22.423029 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.423013 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg"] Sep 16 07:34:22.423140 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.423124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:22.425650 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.425629 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-whsrb\"" Sep 16 07:34:22.425752 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.425655 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.425834 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.425808 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 16 07:34:22.426087 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.426071 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.445097 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.445081 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns"] Sep 16 07:34:22.445193 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.445108 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-vfmqh"] Sep 16 07:34:22.445264 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.445191 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.448202 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.447877 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 16 07:34:22.448202 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.447892 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 16 07:34:22.448202 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.447902 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.448202 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.447907 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.448202 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.447911 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tmr9w\"" Sep 16 07:34:22.467218 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.467199 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4"] Sep 16 07:34:22.467346 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.467329 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.471808 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.471785 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 16 07:34:22.472318 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.472059 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 16 07:34:22.472318 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.472299 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7l5hj\"" Sep 16 07:34:22.472481 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.472301 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.473228 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.473195 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.480315 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.480298 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 16 07:34:22.482585 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.482564 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-trusted-ca\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.482681 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.482602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.482681 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.482623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-installation-pull-secrets\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.482681 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.482644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-image-registry-private-configuration\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.482681 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.482666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-bound-sa-token\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.482895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.482710 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-ca-trust-extracted\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.482895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.482805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98xn\" (UniqueName: \"kubernetes.io/projected/7b73b576-b7d6-46ec-a73c-d6cdc1ec0fc7-kube-api-access-s98xn\") pod \"volume-data-source-validator-86695b7cc7-p8wns\" (UID: \"7b73b576-b7d6-46ec-a73c-d6cdc1ec0fc7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns" Sep 16 07:34:22.482895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.482854 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-certificates\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.482895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.482888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtvpg\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-kube-api-access-jtvpg\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.491605 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.491587 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-d5hqp"] Sep 16 07:34:22.491707 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.491688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.494593 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.494572 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.494695 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.494576 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 16 07:34:22.494764 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.494730 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2jx6m\"" Sep 16 07:34:22.495015 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.494998 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.495111 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.495024 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 16 07:34:22.515451 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.515431 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-58f8796bd8-45bdh"] Sep 16 07:34:22.515557 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.515543 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.518284 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.518264 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 16 07:34:22.519003 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.518980 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.519124 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.518987 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.519422 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.519403 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 16 07:34:22.519532 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.519454 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x6hfs\"" Sep 16 07:34:22.528041 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.528019 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 16 07:34:22.539341 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.539319 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq"] Sep 16 07:34:22.539437 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.539423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.542090 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.542073 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 16 07:34:22.542307 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.542291 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4h2j4\"" Sep 16 07:34:22.543068 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.542633 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 16 07:34:22.543068 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.542655 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 16 07:34:22.543068 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.542696 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 16 07:34:22.543068 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.542889 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.543068 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.542940 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.559659 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.559636 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2pxc6"] Sep 16 07:34:22.559784 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.559742 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:22.559851 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.559832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:22.562576 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.562473 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zf7hx\"" Sep 16 07:34:22.562576 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.562471 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.562980 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.562956 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 16 07:34:22.563072 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.563045 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 07:34:22.563072 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.563063 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 16 07:34:22.563187 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.563156 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k8bpx\"" Sep 16 07:34:22.563299 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.563279 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.575499 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.575477 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79ffc6588-khnvp"] Sep 16 07:34:22.575635 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.575515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9"] Sep 16 07:34:22.575635 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.575546 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-vfmqh"] Sep 16 07:34:22.575635 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.575564 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-d5hqp"] Sep 16 07:34:22.575635 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.575577 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2pxc6"] Sep 16 07:34:22.575635 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.575599 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x"] Sep 16 07:34:22.575635 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.575612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.578258 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.578239 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 07:34:22.578367 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.578350 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2msf4\"" Sep 16 07:34:22.578511 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.578489 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 07:34:22.583189 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s98xn\" (UniqueName: \"kubernetes.io/projected/7b73b576-b7d6-46ec-a73c-d6cdc1ec0fc7-kube-api-access-s98xn\") pod \"volume-data-source-validator-86695b7cc7-p8wns\" (UID: \"7b73b576-b7d6-46ec-a73c-d6cdc1ec0fc7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns" Sep 16 07:34:22.583291 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-serving-cert\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.583291 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-certificates\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.583410 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtvpg\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-kube-api-access-jtvpg\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.583410 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-snapshots\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.583410 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-service-ca-bundle\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.583410 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-trusted-ca\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.583605 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-trusted-ca-bundle\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.583605 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:22.583605 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.583605 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583519 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d426f\" (UniqueName: \"kubernetes.io/projected/6f8ff03a-7366-4924-a639-ec3d9e63c398-kube-api-access-d426f\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9rgl4\" (UID: \"6f8ff03a-7366-4924-a639-ec3d9e63c398\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.583793 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.583607 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:22.583793 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.583621 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79ffc6588-khnvp: secret "image-registry-tls" not found Sep 16 07:34:22.583793 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b-serving-cert\") pod \"service-ca-operator-cf7c776b7-btmxg\" (UID: \"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.583793 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.583680 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls podName:dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa nodeName:}" failed. No retries permitted until 2026-09-16 07:34:23.083660524 +0000 UTC m=+36.003276461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls") pod "image-registry-79ffc6588-khnvp" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa") : secret "image-registry-tls" not found Sep 16 07:34:22.583793 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkds5\" (UniqueName: \"kubernetes.io/projected/f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b-kube-api-access-kkds5\") pod \"service-ca-operator-cf7c776b7-btmxg\" (UID: \"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.583793 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f8ff03a-7366-4924-a639-ec3d9e63c398-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9rgl4\" (UID: \"6f8ff03a-7366-4924-a639-ec3d9e63c398\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.583793 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-installation-pull-secrets\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.583793 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-tmp\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.584144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2qq\" (UniqueName: \"kubernetes.io/projected/8d486908-d637-4137-a9b1-f47a46e2aeac-kube-api-access-nw2qq\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:22.584144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-image-registry-private-configuration\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.584144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-bound-sa-token\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.584144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4lg7\" (UniqueName: \"kubernetes.io/projected/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-kube-api-access-b4lg7\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.584144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-ca-trust-extracted\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.584144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f8ff03a-7366-4924-a639-ec3d9e63c398-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9rgl4\" (UID: \"6f8ff03a-7366-4924-a639-ec3d9e63c398\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.584144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-certificates\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.584144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.583972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b-config\") pod \"service-ca-operator-cf7c776b7-btmxg\" (UID: \"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.584512 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.584238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-ca-trust-extracted\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.584884 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.584861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-trusted-ca\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.588326 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.588308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-installation-pull-secrets\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.588326 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.588317 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-image-registry-private-configuration\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.592830 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.592711 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98xn\" (UniqueName: \"kubernetes.io/projected/7b73b576-b7d6-46ec-a73c-d6cdc1ec0fc7-kube-api-access-s98xn\") pod \"volume-data-source-validator-86695b7cc7-p8wns\" (UID: \"7b73b576-b7d6-46ec-a73c-d6cdc1ec0fc7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns" Sep 16 07:34:22.593992 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.593975 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-85jjc"] Sep 16 07:34:22.594093 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.594076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:22.596042 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.596019 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-bound-sa-token\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.596368 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.596347 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtvpg\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-kube-api-access-jtvpg\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:22.596546 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.596527 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 16 07:34:22.596610 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.596576 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 16 07:34:22.596653 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.596637 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-str4h\"" Sep 16 07:34:22.620071 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.620054 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg"] Sep 16 07:34:22.620163 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.620153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-85jjc" Sep 16 07:34:22.622854 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.622825 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mh2l7\"" Sep 16 07:34:22.622854 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.622831 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.623006 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.622867 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.623006 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.622868 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 07:34:22.630195 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.630178 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58f8796bd8-45bdh"] Sep 16 07:34:22.630280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.630203 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4"] Sep 16 07:34:22.630280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.630232 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg"] Sep 16 07:34:22.630280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.630247 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x"] Sep 16 07:34:22.630280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.630259 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-85jjc"] Sep 16 07:34:22.630280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.630270 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg"] Sep 16 07:34:22.630280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.630271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg" Sep 16 07:34:22.630280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.630282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq"] Sep 16 07:34:22.632529 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.632514 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.632612 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.632582 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.632612 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.632599 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-p6nhq\"" Sep 16 07:34:22.684403 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684385 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwkg\" (UniqueName: \"kubernetes.io/projected/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-kube-api-access-2wwkg\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:22.684509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:22.684509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d785aaa-20ec-484a-a81e-48247bc83110-tmp-dir\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.684509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f8ff03a-7366-4924-a639-ec3d9e63c398-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9rgl4\" (UID: \"6f8ff03a-7366-4924-a639-ec3d9e63c398\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.684509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-snapshots\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.684509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-service-ca-bundle\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.684753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:22.684753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkds5\" (UniqueName: \"kubernetes.io/projected/f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b-kube-api-access-kkds5\") pod \"service-ca-operator-cf7c776b7-btmxg\" (UID: \"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.684753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-serving-cert\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.684753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-stats-auth\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.684753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3ad689-9905-4ae0-8ad1-0364dea219fe-serving-cert\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.684753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:22.685036 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.685036 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:22.685036 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-trusted-ca-bundle\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.685036 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.684921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f8ff03a-7366-4924-a639-ec3d9e63c398-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9rgl4\" (UID: \"6f8ff03a-7366-4924-a639-ec3d9e63c398\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.685241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-snapshots\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.685241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-tmp\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.685241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-service-ca-bundle\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.685241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3ad689-9905-4ae0-8ad1-0364dea219fe-config\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.685447 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.685447 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lg7\" (UniqueName: \"kubernetes.io/projected/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-kube-api-access-b4lg7\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.685447 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b-config\") pod \"service-ca-operator-cf7c776b7-btmxg\" (UID: \"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.685567 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685447 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-tmp\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.685567 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685455 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f8ff03a-7366-4924-a639-ec3d9e63c398-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9rgl4\" (UID: \"6f8ff03a-7366-4924-a639-ec3d9e63c398\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.685660 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d426f\" (UniqueName: \"kubernetes.io/projected/6f8ff03a-7366-4924-a639-ec3d9e63c398-kube-api-access-d426f\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9rgl4\" (UID: \"6f8ff03a-7366-4924-a639-ec3d9e63c398\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.685708 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.685708 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57w4p\" (UniqueName: \"kubernetes.io/projected/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-kube-api-access-57w4p\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.685799 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ad689-9905-4ae0-8ad1-0364dea219fe-trusted-ca\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.685799 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685753 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spqnj\" (UniqueName: \"kubernetes.io/projected/0a3ad689-9905-4ae0-8ad1-0364dea219fe-kube-api-access-spqnj\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.685799 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-default-certificate\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.685944 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d785aaa-20ec-484a-a81e-48247bc83110-config-volume\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.685944 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:22.685944 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b-serving-cert\") pod \"service-ca-operator-cf7c776b7-btmxg\" (UID: \"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.686084 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b-config\") pod \"service-ca-operator-cf7c776b7-btmxg\" (UID: \"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.686084 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.685992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67xkv\" (UniqueName: \"kubernetes.io/projected/3d785aaa-20ec-484a-a81e-48247bc83110-kube-api-access-67xkv\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.686084 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.686021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-trusted-ca-bundle\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.686084 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.686060 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 07:34:22.686290 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.686097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2qq\" (UniqueName: \"kubernetes.io/projected/8d486908-d637-4137-a9b1-f47a46e2aeac-kube-api-access-nw2qq\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:22.686350 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.686306 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls podName:8d486908-d637-4137-a9b1-f47a46e2aeac nodeName:}" failed. No retries permitted until 2026-09-16 07:34:23.186285592 +0000 UTC m=+36.105901527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dwzp9" (UID: "8d486908-d637-4137-a9b1-f47a46e2aeac") : secret "samples-operator-tls" not found Sep 16 07:34:22.687266 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.687239 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f8ff03a-7366-4924-a639-ec3d9e63c398-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9rgl4\" (UID: \"6f8ff03a-7366-4924-a639-ec3d9e63c398\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.687388 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.687368 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-serving-cert\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.688019 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.688003 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b-serving-cert\") pod \"service-ca-operator-cf7c776b7-btmxg\" (UID: \"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.702363 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.702345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2qq\" (UniqueName: \"kubernetes.io/projected/8d486908-d637-4137-a9b1-f47a46e2aeac-kube-api-access-nw2qq\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:22.705220 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.705190 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns" Sep 16 07:34:22.706085 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.706066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d426f\" (UniqueName: \"kubernetes.io/projected/6f8ff03a-7366-4924-a639-ec3d9e63c398-kube-api-access-d426f\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9rgl4\" (UID: \"6f8ff03a-7366-4924-a639-ec3d9e63c398\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.706337 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.706320 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lg7\" (UniqueName: \"kubernetes.io/projected/3a8af6d2-1836-4dc6-81c3-85fc2466ba2d-kube-api-access-b4lg7\") pod \"insights-operator-5844b856db-vfmqh\" (UID: \"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d\") " pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.707156 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.707134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkds5\" (UniqueName: \"kubernetes.io/projected/f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b-kube-api-access-kkds5\") pod \"service-ca-operator-cf7c776b7-btmxg\" (UID: \"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.754138 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.754113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" Sep 16 07:34:22.777926 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.777908 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-vfmqh" Sep 16 07:34:22.787634 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tl7f\" (UniqueName: \"kubernetes.io/projected/2b68f2e1-cd30-4b92-9984-e2e55252cfef-kube-api-access-6tl7f\") pod \"network-check-source-6f8b768455-7vvrg\" (UID: \"2b68f2e1-cd30-4b92-9984-e2e55252cfef\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg" Sep 16 07:34:22.787634 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67xkv\" (UniqueName: \"kubernetes.io/projected/3d785aaa-20ec-484a-a81e-48247bc83110-kube-api-access-67xkv\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.787634 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwkg\" (UniqueName: \"kubernetes.io/projected/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-kube-api-access-2wwkg\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:22.787920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:22.787920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d785aaa-20ec-484a-a81e-48247bc83110-tmp-dir\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.787920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:22.787920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b-cert\") pod \"ingress-canary-85jjc\" (UID: \"0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b\") " pod="openshift-ingress-canary/ingress-canary-85jjc" Sep 16 07:34:22.787920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-stats-auth\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.787920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3ad689-9905-4ae0-8ad1-0364dea219fe-serving-cert\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.787920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:22.787920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.788470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:22.788470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.787978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzjx5\" (UniqueName: \"kubernetes.io/projected/0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b-kube-api-access-pzjx5\") pod \"ingress-canary-85jjc\" (UID: \"0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b\") " pod="openshift-ingress-canary/ingress-canary-85jjc" Sep 16 07:34:22.788470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.788009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3ad689-9905-4ae0-8ad1-0364dea219fe-config\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.788470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.788040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.788470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.788098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.788470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.788126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57w4p\" (UniqueName: \"kubernetes.io/projected/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-kube-api-access-57w4p\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.788470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.788158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ad689-9905-4ae0-8ad1-0364dea219fe-trusted-ca\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.788470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.788184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spqnj\" (UniqueName: \"kubernetes.io/projected/0a3ad689-9905-4ae0-8ad1-0364dea219fe-kube-api-access-spqnj\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.788470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.788237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-default-certificate\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.788470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.788271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d785aaa-20ec-484a-a81e-48247bc83110-config-volume\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.788934 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.788712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:22.788934 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.788848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d785aaa-20ec-484a-a81e-48247bc83110-config-volume\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.789030 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.789104 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls podName:5cfd20db-3fe7-47a1-bb11-8a8421cd83fb nodeName:}" failed. No retries permitted until 2026-09-16 07:34:23.289075144 +0000 UTC m=+36.208691089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sppbq" (UID: "5cfd20db-3fe7-47a1-bb11-8a8421cd83fb") : secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.789496 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d785aaa-20ec-484a-a81e-48247bc83110-tmp-dir\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.789589 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.789632 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls podName:3d785aaa-20ec-484a-a81e-48247bc83110 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:23.2896165 +0000 UTC m=+36.209232438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls") pod "dns-default-2pxc6" (UID: "3d785aaa-20ec-484a-a81e-48247bc83110") : secret "dns-default-metrics-tls" not found Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.789778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3ad689-9905-4ae0-8ad1-0364dea219fe-config\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.790242 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.790269 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:23.29025264 +0000 UTC m=+36.209868588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : configmap references non-existent config key: service-ca.crt Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.790333 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert podName:6c06db71-a2bf-4b00-82f2-3c1bc21555a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:23.290314995 +0000 UTC m=+36.209930920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cvm5x" (UID: "6c06db71-a2bf-4b00-82f2-3c1bc21555a0") : secret "networking-console-plugin-cert" not found Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.790392 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:22.790429 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:23.290417507 +0000 UTC m=+36.210033634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : secret "router-metrics-certs-default" not found Sep 16 07:34:22.790547 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.790495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:22.791599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.791548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ad689-9905-4ae0-8ad1-0364dea219fe-trusted-ca\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.793144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.793023 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-stats-auth\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.793874 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.793851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-default-certificate\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.794159 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.794133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3ad689-9905-4ae0-8ad1-0364dea219fe-serving-cert\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.798969 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.798927 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwkg\" (UniqueName: \"kubernetes.io/projected/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-kube-api-access-2wwkg\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:22.799605 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.799560 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67xkv\" (UniqueName: \"kubernetes.io/projected/3d785aaa-20ec-484a-a81e-48247bc83110-kube-api-access-67xkv\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:22.800980 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.800944 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" Sep 16 07:34:22.801499 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.801470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57w4p\" (UniqueName: \"kubernetes.io/projected/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-kube-api-access-57w4p\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:22.814134 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.814111 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spqnj\" (UniqueName: \"kubernetes.io/projected/0a3ad689-9905-4ae0-8ad1-0364dea219fe-kube-api-access-spqnj\") pod \"console-operator-696884fb9c-d5hqp\" (UID: \"0a3ad689-9905-4ae0-8ad1-0364dea219fe\") " pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.826437 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.826403 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:22.889604 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.889147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b-cert\") pod \"ingress-canary-85jjc\" (UID: \"0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b\") " pod="openshift-ingress-canary/ingress-canary-85jjc" Sep 16 07:34:22.889604 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.889275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzjx5\" (UniqueName: \"kubernetes.io/projected/0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b-kube-api-access-pzjx5\") pod \"ingress-canary-85jjc\" (UID: \"0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b\") " pod="openshift-ingress-canary/ingress-canary-85jjc" Sep 16 07:34:22.889604 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.889385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tl7f\" (UniqueName: \"kubernetes.io/projected/2b68f2e1-cd30-4b92-9984-e2e55252cfef-kube-api-access-6tl7f\") pod \"network-check-source-6f8b768455-7vvrg\" (UID: \"2b68f2e1-cd30-4b92-9984-e2e55252cfef\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg" Sep 16 07:34:22.891765 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.891743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b-cert\") pod \"ingress-canary-85jjc\" (UID: \"0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b\") " pod="openshift-ingress-canary/ingress-canary-85jjc" Sep 16 07:34:22.897610 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.897587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tl7f\" (UniqueName: \"kubernetes.io/projected/2b68f2e1-cd30-4b92-9984-e2e55252cfef-kube-api-access-6tl7f\") pod \"network-check-source-6f8b768455-7vvrg\" (UID: \"2b68f2e1-cd30-4b92-9984-e2e55252cfef\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg" Sep 16 07:34:22.897685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.897661 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzjx5\" (UniqueName: \"kubernetes.io/projected/0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b-kube-api-access-pzjx5\") pod \"ingress-canary-85jjc\" (UID: \"0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b\") " pod="openshift-ingress-canary/ingress-canary-85jjc" Sep 16 07:34:22.927666 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.927647 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-85jjc" Sep 16 07:34:22.938399 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:22.938383 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg" Sep 16 07:34:23.012974 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.012942 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns"] Sep 16 07:34:23.043814 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.043757 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg"] Sep 16 07:34:23.048067 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:23.048045 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b1b2ad_ea4e_48bb_a17d_10a2df9cf74b.slice/crio-8194df7a2467df777a1ff9b4445e0e35c9579ee7345d6d97f848ee41d4c3d047 WatchSource:0}: Error finding container 8194df7a2467df777a1ff9b4445e0e35c9579ee7345d6d97f848ee41d4c3d047: Status 404 returned error can't find the container with id 8194df7a2467df777a1ff9b4445e0e35c9579ee7345d6d97f848ee41d4c3d047 Sep 16 07:34:23.084027 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.084006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-vfmqh"] Sep 16 07:34:23.086171 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:23.086146 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8af6d2_1836_4dc6_81c3_85fc2466ba2d.slice/crio-ba5455d57cc61d0b525671828ffc9d181be900168d656ec37f38d50ab2642832 WatchSource:0}: Error finding container ba5455d57cc61d0b525671828ffc9d181be900168d656ec37f38d50ab2642832: Status 404 returned error can't find the container with id ba5455d57cc61d0b525671828ffc9d181be900168d656ec37f38d50ab2642832 Sep 16 07:34:23.091110 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.091091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:23.091268 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.091252 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:23.091326 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.091269 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79ffc6588-khnvp: secret "image-registry-tls" not found Sep 16 07:34:23.091326 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.091313 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls podName:dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa nodeName:}" failed. No retries permitted until 2026-09-16 07:34:24.091298843 +0000 UTC m=+37.010914766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls") pod "image-registry-79ffc6588-khnvp" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa") : secret "image-registry-tls" not found Sep 16 07:34:23.097970 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.097951 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4"] Sep 16 07:34:23.099936 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:23.099916 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8ff03a_7366_4924_a639_ec3d9e63c398.slice/crio-d300e1bb8c0cd74573ae18bed208c6d8c93a24473c04f07527cdbddf5d2cf7d5 WatchSource:0}: Error finding container d300e1bb8c0cd74573ae18bed208c6d8c93a24473c04f07527cdbddf5d2cf7d5: Status 404 returned error can't find the container with id d300e1bb8c0cd74573ae18bed208c6d8c93a24473c04f07527cdbddf5d2cf7d5 Sep 16 07:34:23.192109 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.192083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:23.192227 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.192195 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 07:34:23.192278 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.192272 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls podName:8d486908-d637-4137-a9b1-f47a46e2aeac nodeName:}" failed. No retries permitted until 2026-09-16 07:34:24.192257203 +0000 UTC m=+37.111873126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dwzp9" (UID: "8d486908-d637-4137-a9b1-f47a46e2aeac") : secret "samples-operator-tls" not found Sep 16 07:34:23.222946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.222925 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg"] Sep 16 07:34:23.225764 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.225743 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-85jjc"] Sep 16 07:34:23.239375 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.239352 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-d5hqp"] Sep 16 07:34:23.248129 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:23.248096 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b68f2e1_cd30_4b92_9984_e2e55252cfef.slice/crio-8608c918f7f0d4179f8a16d5b6e81b717077961c34d411c8e5859f1306acd7e7 WatchSource:0}: Error finding container 8608c918f7f0d4179f8a16d5b6e81b717077961c34d411c8e5859f1306acd7e7: Status 404 returned error can't find the container with id 8608c918f7f0d4179f8a16d5b6e81b717077961c34d411c8e5859f1306acd7e7 Sep 16 07:34:23.248404 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:23.248384 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3ad689_9905_4ae0_8ad1_0364dea219fe.slice/crio-4d6acfdee5f15cff1cacdfd96e2e771e529a6d7a88a073ad9774e901abb8449b WatchSource:0}: Error finding container 4d6acfdee5f15cff1cacdfd96e2e771e529a6d7a88a073ad9774e901abb8449b: Status 404 returned error can't find the container with id 4d6acfdee5f15cff1cacdfd96e2e771e529a6d7a88a073ad9774e901abb8449b Sep 16 07:34:23.248652 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:23.248629 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6f5a2d_a0c2_47bd_886c_7a68b8c9bd9b.slice/crio-f00a153441197dffffba4ff12a994f1a7d2e1a59bc5f1f8bac6b323a02a5bd58 WatchSource:0}: Error finding container f00a153441197dffffba4ff12a994f1a7d2e1a59bc5f1f8bac6b323a02a5bd58: Status 404 returned error can't find the container with id f00a153441197dffffba4ff12a994f1a7d2e1a59bc5f1f8bac6b323a02a5bd58 Sep 16 07:34:23.293097 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.293072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:23.293168 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.293127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:23.293168 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.293159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:23.293264 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.293183 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:23.293264 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.293220 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:23.293364 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.293242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:23.293364 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.293283 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls podName:5cfd20db-3fe7-47a1-bb11-8a8421cd83fb nodeName:}" failed. No retries permitted until 2026-09-16 07:34:24.293262991 +0000 UTC m=+37.212878932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sppbq" (UID: "5cfd20db-3fe7-47a1-bb11-8a8421cd83fb") : secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:23.293364 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.293310 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 07:34:23.293364 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.293328 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:23.293364 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.293356 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:24.293342022 +0000 UTC m=+37.212957965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : configmap references non-existent config key: service-ca.crt Sep 16 07:34:23.293539 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.293335 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 07:34:23.293539 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.293400 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert podName:6c06db71-a2bf-4b00-82f2-3c1bc21555a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:24.293393702 +0000 UTC m=+37.213009626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cvm5x" (UID: "6c06db71-a2bf-4b00-82f2-3c1bc21555a0") : secret "networking-console-plugin-cert" not found Sep 16 07:34:23.293539 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.293412 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls podName:3d785aaa-20ec-484a-a81e-48247bc83110 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:24.293406182 +0000 UTC m=+37.213022105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls") pod "dns-default-2pxc6" (UID: "3d785aaa-20ec-484a-a81e-48247bc83110") : secret "dns-default-metrics-tls" not found Sep 16 07:34:23.293539 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:23.293427 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:24.293419587 +0000 UTC m=+37.213035512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : secret "router-metrics-certs-default" not found Sep 16 07:34:23.554320 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.554293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:23.555018 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.554443 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:23.557526 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.557370 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lz8hv\"" Sep 16 07:34:23.557621 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.557371 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 07:34:23.723456 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.723417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg" event={"ID":"2b68f2e1-cd30-4b92-9984-e2e55252cfef","Type":"ContainerStarted","Data":"8608c918f7f0d4179f8a16d5b6e81b717077961c34d411c8e5859f1306acd7e7"} Sep 16 07:34:23.725199 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.725170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-85jjc" event={"ID":"0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b","Type":"ContainerStarted","Data":"f00a153441197dffffba4ff12a994f1a7d2e1a59bc5f1f8bac6b323a02a5bd58"} Sep 16 07:34:23.726717 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.726691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vfmqh" event={"ID":"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d","Type":"ContainerStarted","Data":"ba5455d57cc61d0b525671828ffc9d181be900168d656ec37f38d50ab2642832"} Sep 16 07:34:23.728282 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.728253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns" event={"ID":"7b73b576-b7d6-46ec-a73c-d6cdc1ec0fc7","Type":"ContainerStarted","Data":"1cc0002f240fcc40994f96c9bafc0b92021b81aa2faaace811ee83eadcac662f"} Sep 16 07:34:23.729705 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.729655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" event={"ID":"6f8ff03a-7366-4924-a639-ec3d9e63c398","Type":"ContainerStarted","Data":"d300e1bb8c0cd74573ae18bed208c6d8c93a24473c04f07527cdbddf5d2cf7d5"} Sep 16 07:34:23.733472 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.733348 2580 generic.go:356] "Generic (PLEG): container finished" podID="05675886-b0f7-47ae-b1ee-355badf36fff" containerID="b0b908ab332fd3f82071d1d5fa54986754e391e5fbc03b451a4843b0be5b1e7b" exitCode=0 Sep 16 07:34:23.733472 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.733430 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pl84" event={"ID":"05675886-b0f7-47ae-b1ee-355badf36fff","Type":"ContainerDied","Data":"b0b908ab332fd3f82071d1d5fa54986754e391e5fbc03b451a4843b0be5b1e7b"} Sep 16 07:34:23.736493 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.736468 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" event={"ID":"0a3ad689-9905-4ae0-8ad1-0364dea219fe","Type":"ContainerStarted","Data":"4d6acfdee5f15cff1cacdfd96e2e771e529a6d7a88a073ad9774e901abb8449b"} Sep 16 07:34:23.737992 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:23.737961 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" event={"ID":"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b","Type":"ContainerStarted","Data":"8194df7a2467df777a1ff9b4445e0e35c9579ee7345d6d97f848ee41d4c3d047"} Sep 16 07:34:24.102177 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:24.101406 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:24.102177 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.101662 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:24.102177 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.101681 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79ffc6588-khnvp: secret "image-registry-tls" not found Sep 16 07:34:24.102177 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.101742 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls podName:dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa nodeName:}" failed. No retries permitted until 2026-09-16 07:34:26.101724343 +0000 UTC m=+39.021340284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls") pod "image-registry-79ffc6588-khnvp" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa") : secret "image-registry-tls" not found Sep 16 07:34:24.203389 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:24.202659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:24.203389 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.202955 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 07:34:24.203389 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.203019 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls podName:8d486908-d637-4137-a9b1-f47a46e2aeac nodeName:}" failed. No retries permitted until 2026-09-16 07:34:26.203000946 +0000 UTC m=+39.122616887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dwzp9" (UID: "8d486908-d637-4137-a9b1-f47a46e2aeac") : secret "samples-operator-tls" not found Sep 16 07:34:24.304292 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:24.303519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:24.304292 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:24.303578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:24.304292 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:24.303621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:24.304292 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:24.303673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:24.304292 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:24.303760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:24.304292 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.303894 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:24.304292 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.303958 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls podName:5cfd20db-3fe7-47a1-bb11-8a8421cd83fb nodeName:}" failed. No retries permitted until 2026-09-16 07:34:26.303939862 +0000 UTC m=+39.223555790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sppbq" (UID: "5cfd20db-3fe7-47a1-bb11-8a8421cd83fb") : secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:24.304780 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.304350 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:24.304780 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.304374 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:26.304355838 +0000 UTC m=+39.223971780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : configmap references non-existent config key: service-ca.crt Sep 16 07:34:24.304780 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.304414 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls podName:3d785aaa-20ec-484a-a81e-48247bc83110 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:26.304396825 +0000 UTC m=+39.224012764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls") pod "dns-default-2pxc6" (UID: "3d785aaa-20ec-484a-a81e-48247bc83110") : secret "dns-default-metrics-tls" not found Sep 16 07:34:24.304780 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.304458 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 07:34:24.304780 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.304526 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 07:34:24.304780 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.304528 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert podName:6c06db71-a2bf-4b00-82f2-3c1bc21555a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:26.304513103 +0000 UTC m=+39.224129046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cvm5x" (UID: "6c06db71-a2bf-4b00-82f2-3c1bc21555a0") : secret "networking-console-plugin-cert" not found Sep 16 07:34:24.304780 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:24.304585 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:26.30456742 +0000 UTC m=+39.224183350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : secret "router-metrics-certs-default" not found Sep 16 07:34:24.749769 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:24.749731 2580 generic.go:356] "Generic (PLEG): container finished" podID="05675886-b0f7-47ae-b1ee-355badf36fff" containerID="7b92a2a15857bba43429fdf68e1055b637eaa958dee239496a0f4d8e99d18ffa" exitCode=0 Sep 16 07:34:24.750256 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:24.749809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pl84" event={"ID":"05675886-b0f7-47ae-b1ee-355badf36fff","Type":"ContainerDied","Data":"7b92a2a15857bba43429fdf68e1055b637eaa958dee239496a0f4d8e99d18ffa"} Sep 16 07:34:25.756048 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:25.755780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns" event={"ID":"7b73b576-b7d6-46ec-a73c-d6cdc1ec0fc7","Type":"ContainerStarted","Data":"a0d702bcf7743b540cc6f7dff438cad7e4a5c8bbcb7884b4c7bd12c91c6a0b71"} Sep 16 07:34:25.761775 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:25.761742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pl84" event={"ID":"05675886-b0f7-47ae-b1ee-355badf36fff","Type":"ContainerStarted","Data":"8acdcf348fb43be94da30adf8288ba4468218c3769746a1aa9a41dfd53a215c3"} Sep 16 07:34:25.773285 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:25.773236 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-p8wns" podStartSLOduration=32.75622089 podStartE2EDuration="34.773203375s" totalImagesPullingTime="2.016982485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:25.772442013 +0000 UTC m=+38.692057959" watchObservedRunningTime="2026-09-16 07:34:25.773203375 +0000 UTC m=+38.692819322" Sep 16 07:34:25.796683 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:25.796274 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6pl84" podStartSLOduration=9.842644051 podStartE2EDuration="37.796255721s" totalImagesPullingTime="27.95361167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:25.795058059 +0000 UTC m=+38.714674020" watchObservedRunningTime="2026-09-16 07:34:25.796255721 +0000 UTC m=+38.715871670" Sep 16 07:34:26.124714 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:26.124629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:26.124852 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.124781 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:26.124852 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.124805 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79ffc6588-khnvp: secret "image-registry-tls" not found Sep 16 07:34:26.124942 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.124873 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls podName:dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa nodeName:}" failed. No retries permitted until 2026-09-16 07:34:30.124852985 +0000 UTC m=+43.044468912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls") pod "image-registry-79ffc6588-khnvp" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa") : secret "image-registry-tls" not found Sep 16 07:34:26.225452 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:26.225411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:26.225597 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.225561 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 07:34:26.225647 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.225630 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls podName:8d486908-d637-4137-a9b1-f47a46e2aeac nodeName:}" failed. No retries permitted until 2026-09-16 07:34:30.225613442 +0000 UTC m=+43.145229366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dwzp9" (UID: "8d486908-d637-4137-a9b1-f47a46e2aeac") : secret "samples-operator-tls" not found Sep 16 07:34:26.326836 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:26.326796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:26.327023 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:26.326973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:26.327091 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:26.327036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:26.327091 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:26.327076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:26.327200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:26.327123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:26.327922 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.327374 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 07:34:26.327922 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.327393 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:30.327375873 +0000 UTC m=+43.246991815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : configmap references non-existent config key: service-ca.crt Sep 16 07:34:26.327922 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.327453 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert podName:6c06db71-a2bf-4b00-82f2-3c1bc21555a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:30.327440396 +0000 UTC m=+43.247056322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cvm5x" (UID: "6c06db71-a2bf-4b00-82f2-3c1bc21555a0") : secret "networking-console-plugin-cert" not found Sep 16 07:34:26.327922 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.327479 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:26.327922 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.327513 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls podName:5cfd20db-3fe7-47a1-bb11-8a8421cd83fb nodeName:}" failed. No retries permitted until 2026-09-16 07:34:30.3275012 +0000 UTC m=+43.247117133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sppbq" (UID: "5cfd20db-3fe7-47a1-bb11-8a8421cd83fb") : secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:26.327922 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.327522 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:26.327922 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.327566 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 07:34:26.327922 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.327587 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls podName:3d785aaa-20ec-484a-a81e-48247bc83110 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:30.32757718 +0000 UTC m=+43.247193120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls") pod "dns-default-2pxc6" (UID: "3d785aaa-20ec-484a-a81e-48247bc83110") : secret "dns-default-metrics-tls" not found Sep 16 07:34:26.327922 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:26.327603 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:30.327595144 +0000 UTC m=+43.247211070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : secret "router-metrics-certs-default" not found Sep 16 07:34:30.163478 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:30.163441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:30.163888 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.163614 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:30.163888 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.163639 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79ffc6588-khnvp: secret "image-registry-tls" not found Sep 16 07:34:30.163888 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.163713 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls podName:dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa nodeName:}" failed. No retries permitted until 2026-09-16 07:34:38.163693072 +0000 UTC m=+51.083309002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls") pod "image-registry-79ffc6588-khnvp" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa") : secret "image-registry-tls" not found Sep 16 07:34:30.264550 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:30.264513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:30.264719 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.264667 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 07:34:30.264793 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.264730 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls podName:8d486908-d637-4137-a9b1-f47a46e2aeac nodeName:}" failed. No retries permitted until 2026-09-16 07:34:38.264715398 +0000 UTC m=+51.184331321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dwzp9" (UID: "8d486908-d637-4137-a9b1-f47a46e2aeac") : secret "samples-operator-tls" not found Sep 16 07:34:30.365448 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:30.365402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:30.365623 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:30.365459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:30.365623 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:30.365499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:30.365623 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:30.365543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:30.365623 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.365567 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 07:34:30.365832 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.365673 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 07:34:30.365832 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:30.365687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:30.365832 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.365717 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:30.365832 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.365738 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:38.365719207 +0000 UTC m=+51.285335145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : secret "router-metrics-certs-default" not found Sep 16 07:34:30.365832 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.365771 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert podName:6c06db71-a2bf-4b00-82f2-3c1bc21555a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:38.365760394 +0000 UTC m=+51.285376324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cvm5x" (UID: "6c06db71-a2bf-4b00-82f2-3c1bc21555a0") : secret "networking-console-plugin-cert" not found Sep 16 07:34:30.365832 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.365774 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:30.365832 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.365788 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls podName:3d785aaa-20ec-484a-a81e-48247bc83110 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:38.365780781 +0000 UTC m=+51.285396710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls") pod "dns-default-2pxc6" (UID: "3d785aaa-20ec-484a-a81e-48247bc83110") : secret "dns-default-metrics-tls" not found Sep 16 07:34:30.365832 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.365823 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls podName:5cfd20db-3fe7-47a1-bb11-8a8421cd83fb nodeName:}" failed. No retries permitted until 2026-09-16 07:34:38.365808219 +0000 UTC m=+51.285424152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sppbq" (UID: "5cfd20db-3fe7-47a1-bb11-8a8421cd83fb") : secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:30.366228 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:30.365840 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:38.365831407 +0000 UTC m=+51.285447333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : configmap references non-existent config key: service-ca.crt Sep 16 07:34:31.779465 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.779427 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg" event={"ID":"2b68f2e1-cd30-4b92-9984-e2e55252cfef","Type":"ContainerStarted","Data":"8d9812d298caf1121dd55013c72200f78e9d2361c8afbb952cad92d35e9cd4af"} Sep 16 07:34:31.781394 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.781366 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-85jjc" event={"ID":"0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b","Type":"ContainerStarted","Data":"ede94f0c33bb3c96d5487009eee776961b28276eb61c23d978cae0a910cab586"} Sep 16 07:34:31.783139 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.783110 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vfmqh" event={"ID":"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d","Type":"ContainerStarted","Data":"11d304d2bce61982ed3a148e05d7f9c251a4240e686a34ca3a934015620f47fc"} Sep 16 07:34:31.784790 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.784740 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" event={"ID":"6f8ff03a-7366-4924-a639-ec3d9e63c398","Type":"ContainerStarted","Data":"5802b3650512b8c91b4ef7cc302981171fa09b73884c8418fe54f3ef2311565c"} Sep 16 07:34:31.786514 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.786497 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/0.log" Sep 16 07:34:31.786619 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.786531 2580 generic.go:356] "Generic (PLEG): container finished" podID="0a3ad689-9905-4ae0-8ad1-0364dea219fe" containerID="6ec0808b7da51fb2aca773dec9ad6710cf08d15132643dec11c351fdeee04105" exitCode=255 Sep 16 07:34:31.786619 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.786602 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" event={"ID":"0a3ad689-9905-4ae0-8ad1-0364dea219fe","Type":"ContainerDied","Data":"6ec0808b7da51fb2aca773dec9ad6710cf08d15132643dec11c351fdeee04105"} Sep 16 07:34:31.786852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.786833 2580 scope.go:117] "RemoveContainer" containerID="6ec0808b7da51fb2aca773dec9ad6710cf08d15132643dec11c351fdeee04105" Sep 16 07:34:31.789352 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.789331 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" event={"ID":"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b","Type":"ContainerStarted","Data":"e59548f6a9265209b1c1762de9fb7d3da0b885a1311488418586dba4869e4717"} Sep 16 07:34:31.794648 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.794608 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7vvrg" podStartSLOduration=33.038936011 podStartE2EDuration="40.794597546s" totalImagesPullingTime="7.755661535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:31.793725538 +0000 UTC m=+44.713341484" watchObservedRunningTime="2026-09-16 07:34:31.794597546 +0000 UTC m=+44.714213491" Sep 16 07:34:31.854948 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.854875 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-85jjc" podStartSLOduration=2.099161486 podStartE2EDuration="9.854856022s" totalImagesPullingTime="7.755694536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:31.853450045 +0000 UTC m=+44.773065992" watchObservedRunningTime="2026-09-16 07:34:31.854856022 +0000 UTC m=+44.774471969" Sep 16 07:34:31.855189 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.855161 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-vfmqh" podStartSLOduration=32.938177687 podStartE2EDuration="40.855148997s" totalImagesPullingTime="7.91697131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:31.824126923 +0000 UTC m=+44.743742883" watchObservedRunningTime="2026-09-16 07:34:31.855148997 +0000 UTC m=+44.774764943" Sep 16 07:34:31.871238 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.870572 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" podStartSLOduration=32.9607894 podStartE2EDuration="40.870555167s" totalImagesPullingTime="7.909765767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:31.869464135 +0000 UTC m=+44.789080083" watchObservedRunningTime="2026-09-16 07:34:31.870555167 +0000 UTC m=+44.790171118" Sep 16 07:34:31.887591 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:31.887370 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" podStartSLOduration=32.932761744 podStartE2EDuration="40.887351867s" totalImagesPullingTime="7.954590123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:31.885602082 +0000 UTC m=+44.805218029" watchObservedRunningTime="2026-09-16 07:34:31.887351867 +0000 UTC m=+44.806967813" Sep 16 07:34:32.248862 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.248821 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j28rh"] Sep 16 07:34:32.265974 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.265945 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j28rh"] Sep 16 07:34:32.266112 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.266044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.268691 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.268670 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 07:34:32.268826 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.268679 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-57x9m\"" Sep 16 07:34:32.268826 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.268813 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 07:34:32.283835 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.283807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-crio-socket\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.283989 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.283850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-data-volume\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.283989 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.283876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.283989 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.283968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.284160 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.284004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjxd2\" (UniqueName: \"kubernetes.io/projected/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-api-access-hjxd2\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.384749 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.384695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjxd2\" (UniqueName: \"kubernetes.io/projected/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-api-access-hjxd2\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.384864 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.384798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-crio-socket\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.384864 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.384835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-data-volume\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.384970 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.384859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.384970 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.384955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.385068 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.385028 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-crio-socket\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.385068 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:32.385047 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:34:32.385068 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:32.385055 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:34:32.385232 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:32.385110 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm podName:73507fa6-1ed8-49e8-ab6c-6f635b78a0a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:32.885089981 +0000 UTC m=+45.804705919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j28rh" (UID: "73507fa6-1ed8-49e8-ab6c-6f635b78a0a0") : configmap "kube-rbac-proxy" not found Sep 16 07:34:32.385232 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:32.385131 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls podName:73507fa6-1ed8-49e8-ab6c-6f635b78a0a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:32.885120242 +0000 UTC m=+45.804736166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j28rh" (UID: "73507fa6-1ed8-49e8-ab6c-6f635b78a0a0") : secret "insights-runtime-extractor-tls" not found Sep 16 07:34:32.385232 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.385139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-data-volume\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.395145 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.395122 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjxd2\" (UniqueName: \"kubernetes.io/projected/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-api-access-hjxd2\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.794850 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.794822 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:34:32.795296 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.795281 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/0.log" Sep 16 07:34:32.795351 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.795314 2580 generic.go:356] "Generic (PLEG): container finished" podID="0a3ad689-9905-4ae0-8ad1-0364dea219fe" containerID="82bc583bacc6cfbedcbed4a08c0a69738513c8c9a947c2e66a3ecfa73315974a" exitCode=255 Sep 16 07:34:32.795430 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.795407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" event={"ID":"0a3ad689-9905-4ae0-8ad1-0364dea219fe","Type":"ContainerDied","Data":"82bc583bacc6cfbedcbed4a08c0a69738513c8c9a947c2e66a3ecfa73315974a"} Sep 16 07:34:32.795472 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.795465 2580 scope.go:117] "RemoveContainer" containerID="6ec0808b7da51fb2aca773dec9ad6710cf08d15132643dec11c351fdeee04105" Sep 16 07:34:32.795788 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.795651 2580 scope.go:117] "RemoveContainer" containerID="82bc583bacc6cfbedcbed4a08c0a69738513c8c9a947c2e66a3ecfa73315974a" Sep 16 07:34:32.795918 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:32.795807 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-d5hqp_openshift-console-operator(0a3ad689-9905-4ae0-8ad1-0364dea219fe)\"" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" podUID="0a3ad689-9905-4ae0-8ad1-0364dea219fe" Sep 16 07:34:32.826977 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.826956 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:32.827068 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.826999 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:32.889354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.889282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.890125 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:32.889554 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:34:32.890125 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:32.889641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:32.890125 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:32.889814 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls podName:73507fa6-1ed8-49e8-ab6c-6f635b78a0a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:33.889792506 +0000 UTC m=+46.809408437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j28rh" (UID: "73507fa6-1ed8-49e8-ab6c-6f635b78a0a0") : secret "insights-runtime-extractor-tls" not found Sep 16 07:34:32.890125 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:32.889866 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:34:32.890125 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:32.889923 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm podName:73507fa6-1ed8-49e8-ab6c-6f635b78a0a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:33.889911649 +0000 UTC m=+46.809527602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j28rh" (UID: "73507fa6-1ed8-49e8-ab6c-6f635b78a0a0") : configmap "kube-rbac-proxy" not found Sep 16 07:34:33.604638 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:33.604572 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h69ht_5b2f9240-4d84-45e6-bfc3-fa31321bda0c/dns-node-resolver/0.log" Sep 16 07:34:33.799398 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:33.799371 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:34:33.799749 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:33.799735 2580 scope.go:117] "RemoveContainer" containerID="82bc583bacc6cfbedcbed4a08c0a69738513c8c9a947c2e66a3ecfa73315974a" Sep 16 07:34:33.799905 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:33.799887 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-d5hqp_openshift-console-operator(0a3ad689-9905-4ae0-8ad1-0364dea219fe)\"" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" podUID="0a3ad689-9905-4ae0-8ad1-0364dea219fe" Sep 16 07:34:33.897920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:33.897851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:33.898051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:33.897960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:33.898051 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:33.897966 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:34:33.898051 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:33.898041 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm podName:73507fa6-1ed8-49e8-ab6c-6f635b78a0a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:35.89802136 +0000 UTC m=+48.817637302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j28rh" (UID: "73507fa6-1ed8-49e8-ab6c-6f635b78a0a0") : configmap "kube-rbac-proxy" not found Sep 16 07:34:33.898180 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:33.898075 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:34:33.898180 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:33.898120 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls podName:73507fa6-1ed8-49e8-ab6c-6f635b78a0a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:35.898106573 +0000 UTC m=+48.817722504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j28rh" (UID: "73507fa6-1ed8-49e8-ab6c-6f635b78a0a0") : secret "insights-runtime-extractor-tls" not found Sep 16 07:34:34.604704 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:34.604677 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9w78q_36dbd8bf-bd64-4f3c-9d7f-ab618f797a96/node-ca/0.log" Sep 16 07:34:34.803025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:34.802996 2580 scope.go:117] "RemoveContainer" containerID="82bc583bacc6cfbedcbed4a08c0a69738513c8c9a947c2e66a3ecfa73315974a" Sep 16 07:34:34.803393 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:34.803162 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-d5hqp_openshift-console-operator(0a3ad689-9905-4ae0-8ad1-0364dea219fe)\"" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" podUID="0a3ad689-9905-4ae0-8ad1-0364dea219fe" Sep 16 07:34:35.604108 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:35.604074 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-85jjc_0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b/serve-healthcheck-canary/0.log" Sep 16 07:34:35.916162 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:35.916129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:35.916532 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:35.916223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:35.916532 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:35.916284 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:34:35.916532 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:35.916346 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls podName:73507fa6-1ed8-49e8-ab6c-6f635b78a0a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:39.916331844 +0000 UTC m=+52.835947772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j28rh" (UID: "73507fa6-1ed8-49e8-ab6c-6f635b78a0a0") : secret "insights-runtime-extractor-tls" not found Sep 16 07:34:35.916532 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:35.916351 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:34:35.916532 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:35.916401 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm podName:73507fa6-1ed8-49e8-ab6c-6f635b78a0a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:39.916388264 +0000 UTC m=+52.836004188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j28rh" (UID: "73507fa6-1ed8-49e8-ab6c-6f635b78a0a0") : configmap "kube-rbac-proxy" not found Sep 16 07:34:38.236065 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:38.236028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:38.236448 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.236183 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:38.236448 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.236202 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79ffc6588-khnvp: secret "image-registry-tls" not found Sep 16 07:34:38.236448 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.236274 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls podName:dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.236259489 +0000 UTC m=+67.155875413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls") pod "image-registry-79ffc6588-khnvp" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa") : secret "image-registry-tls" not found Sep 16 07:34:38.337175 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:38.337149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:38.337342 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.337322 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 07:34:38.337406 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.337395 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls podName:8d486908-d637-4137-a9b1-f47a46e2aeac nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.33737729 +0000 UTC m=+67.256993221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-dwzp9" (UID: "8d486908-d637-4137-a9b1-f47a46e2aeac") : secret "samples-operator-tls" not found Sep 16 07:34:38.438053 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:38.438024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:38.438175 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:38.438092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:38.438175 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:38.438128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:38.438175 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:38.438164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:38.438175 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.438167 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:38.438350 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.438242 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls podName:5cfd20db-3fe7-47a1-bb11-8a8421cd83fb nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.438226226 +0000 UTC m=+67.357842163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sppbq" (UID: "5cfd20db-3fe7-47a1-bb11-8a8421cd83fb") : secret "cluster-monitoring-operator-tls" not found Sep 16 07:34:38.438350 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:38.438261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:38.438350 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.438265 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 07:34:38.438350 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.438265 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:38.438350 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.438344 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.438328269 +0000 UTC m=+67.357944200 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : configmap references non-existent config key: service-ca.crt Sep 16 07:34:38.438557 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.438286 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 07:34:38.438557 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.438370 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert podName:6c06db71-a2bf-4b00-82f2-3c1bc21555a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.438360799 +0000 UTC m=+67.357976727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cvm5x" (UID: "6c06db71-a2bf-4b00-82f2-3c1bc21555a0") : secret "networking-console-plugin-cert" not found Sep 16 07:34:38.438557 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.438395 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls podName:3d785aaa-20ec-484a-a81e-48247bc83110 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.438383744 +0000 UTC m=+67.357999673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls") pod "dns-default-2pxc6" (UID: "3d785aaa-20ec-484a-a81e-48247bc83110") : secret "dns-default-metrics-tls" not found Sep 16 07:34:38.438557 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:38.438448 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs podName:60e1aabf-d4f3-4f85-83dc-72c8c5c61987 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.438426524 +0000 UTC m=+67.358042463 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs") pod "router-default-58f8796bd8-45bdh" (UID: "60e1aabf-d4f3-4f85-83dc-72c8c5c61987") : secret "router-metrics-certs-default" not found Sep 16 07:34:39.952446 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:39.952411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:39.952797 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:39.952504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:39.952797 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:39.952570 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:34:39.952797 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:34:39.952633 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls podName:73507fa6-1ed8-49e8-ab6c-6f635b78a0a0 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:47.952618126 +0000 UTC m=+60.872234050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j28rh" (UID: "73507fa6-1ed8-49e8-ab6c-6f635b78a0a0") : secret "insights-runtime-extractor-tls" not found Sep 16 07:34:39.953023 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:39.953006 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:47.714887 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:47.714856 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zncgj" Sep 16 07:34:48.018398 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.018309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:48.020431 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.020408 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73507fa6-1ed8-49e8-ab6c-6f635b78a0a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j28rh\" (UID: \"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0\") " pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:48.179542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.179517 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-57x9m\"" Sep 16 07:34:48.186798 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.186782 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j28rh" Sep 16 07:34:48.486484 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.486461 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j28rh"] Sep 16 07:34:48.488310 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:48.488278 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73507fa6_1ed8_49e8_ab6c_6f635b78a0a0.slice/crio-338c77b8628684d6ef4fea79457b788e08f953ecbd91d9900aee3dfc2373080f WatchSource:0}: Error finding container 338c77b8628684d6ef4fea79457b788e08f953ecbd91d9900aee3dfc2373080f: Status 404 returned error can't find the container with id 338c77b8628684d6ef4fea79457b788e08f953ecbd91d9900aee3dfc2373080f Sep 16 07:34:48.553834 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.553813 2580 scope.go:117] "RemoveContainer" containerID="82bc583bacc6cfbedcbed4a08c0a69738513c8c9a947c2e66a3ecfa73315974a" Sep 16 07:34:48.846514 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.846438 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j28rh" event={"ID":"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0","Type":"ContainerStarted","Data":"d649fe3a8cb2535253cdcc789b179311c9775d784d0a8f0e52e980eebcbbc120"} Sep 16 07:34:48.846514 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.846506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j28rh" event={"ID":"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0","Type":"ContainerStarted","Data":"338c77b8628684d6ef4fea79457b788e08f953ecbd91d9900aee3dfc2373080f"} Sep 16 07:34:48.848406 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.848386 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:34:48.848522 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.848445 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" event={"ID":"0a3ad689-9905-4ae0-8ad1-0364dea219fe","Type":"ContainerStarted","Data":"4137177fd73f50c67eaa7d09b9853d7f3aead0e97f0b42bcc4d75f1f71ee97fa"} Sep 16 07:34:48.848736 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.848712 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:48.865066 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:48.865024 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" podStartSLOduration=50.109992967 podStartE2EDuration="57.865010631s" totalImagesPullingTime="7.755017664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:48.864476789 +0000 UTC m=+61.784092736" watchObservedRunningTime="2026-09-16 07:34:48.865010631 +0000 UTC m=+61.784626577" Sep 16 07:34:49.251761 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:49.251733 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-d5hqp" Sep 16 07:34:49.853797 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:49.853762 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j28rh" event={"ID":"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0","Type":"ContainerStarted","Data":"adde5b5c2d718a1d1a93cc7b0a4994d9a4b5a0241237efc8dbb307f4f937cc93"} Sep 16 07:34:50.859235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:50.859139 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j28rh" event={"ID":"73507fa6-1ed8-49e8-ab6c-6f635b78a0a0","Type":"ContainerStarted","Data":"7bd6c771760541760d0cefcb5021bfbb3b2c9ffa3ca8035dcef20fedfbaa74e9"} Sep 16 07:34:50.885701 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:50.885647 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j28rh" podStartSLOduration=16.975599636 podStartE2EDuration="18.885629877s" totalImagesPullingTime="1.910030241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:50.884205985 +0000 UTC m=+63.803821930" watchObservedRunningTime="2026-09-16 07:34:50.885629877 +0000 UTC m=+63.805245850" Sep 16 07:34:53.364596 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.364560 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:53.364975 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.364651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:53.367250 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.367233 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 07:34:53.367299 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.367247 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 07:34:53.377624 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.377588 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533591c0-bcd3-4f1f-9ba5-3eb2effa48a1-metrics-certs\") pod \"network-metrics-daemon-8qm8r\" (UID: \"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1\") " pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:53.377733 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.377653 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b40f3f8d-04c4-4581-b929-fabed41b7079-original-pull-secret\") pod \"global-pull-secret-syncer-l2vjc\" (UID: \"b40f3f8d-04c4-4581-b929-fabed41b7079\") " pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:53.465276 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.465251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6v9\" (UniqueName: \"kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9\") pod \"network-check-target-xsx7w\" (UID: \"720450fe-3414-486b-b860-59fbdb3cfda5\") " pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:53.467337 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.467318 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6v9\" (UniqueName: \"kubernetes.io/projected/720450fe-3414-486b-b860-59fbdb3cfda5-kube-api-access-mj6v9\") pod \"network-check-target-xsx7w\" (UID: \"720450fe-3414-486b-b860-59fbdb3cfda5\") " pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:53.478723 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.478706 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zf7hx\"" Sep 16 07:34:53.486923 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.486902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8qm8r" Sep 16 07:34:53.566383 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.566344 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lz8hv\"" Sep 16 07:34:53.568184 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.568142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2vjc" Sep 16 07:34:53.574701 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.574674 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:53.786329 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.786306 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8qm8r"] Sep 16 07:34:53.788850 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:53.788824 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533591c0_bcd3_4f1f_9ba5_3eb2effa48a1.slice/crio-c334b8ccd5c440d48c2242aecf000366915fb0b784a8c6312ab49d171e078a74 WatchSource:0}: Error finding container c334b8ccd5c440d48c2242aecf000366915fb0b784a8c6312ab49d171e078a74: Status 404 returned error can't find the container with id c334b8ccd5c440d48c2242aecf000366915fb0b784a8c6312ab49d171e078a74 Sep 16 07:34:53.859119 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.859094 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l2vjc"] Sep 16 07:34:53.861569 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:53.861544 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40f3f8d_04c4_4581_b929_fabed41b7079.slice/crio-ec6eaab1bc65b9c064a92b168238c998dd7322efbe5070031a89671ba4a78189 WatchSource:0}: Error finding container ec6eaab1bc65b9c064a92b168238c998dd7322efbe5070031a89671ba4a78189: Status 404 returned error can't find the container with id ec6eaab1bc65b9c064a92b168238c998dd7322efbe5070031a89671ba4a78189 Sep 16 07:34:53.868804 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.868776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8qm8r" event={"ID":"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1","Type":"ContainerStarted","Data":"c334b8ccd5c440d48c2242aecf000366915fb0b784a8c6312ab49d171e078a74"} Sep 16 07:34:53.869929 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.869907 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l2vjc" event={"ID":"b40f3f8d-04c4-4581-b929-fabed41b7079","Type":"ContainerStarted","Data":"ec6eaab1bc65b9c064a92b168238c998dd7322efbe5070031a89671ba4a78189"} Sep 16 07:34:53.893092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:53.893074 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsx7w"] Sep 16 07:34:53.895108 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:53.895089 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720450fe_3414_486b_b860_59fbdb3cfda5.slice/crio-26ac7dbd21f3b1ae5b0f568ffa956b547c8c9e28798b8aaf54ae1c8b6d18c64c WatchSource:0}: Error finding container 26ac7dbd21f3b1ae5b0f568ffa956b547c8c9e28798b8aaf54ae1c8b6d18c64c: Status 404 returned error can't find the container with id 26ac7dbd21f3b1ae5b0f568ffa956b547c8c9e28798b8aaf54ae1c8b6d18c64c Sep 16 07:34:54.272398 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.272374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:54.274562 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.274538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") pod \"image-registry-79ffc6588-khnvp\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:54.373128 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.373105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:54.375160 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.375143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d486908-d637-4137-a9b1-f47a46e2aeac-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-dwzp9\" (UID: \"8d486908-d637-4137-a9b1-f47a46e2aeac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:54.473905 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.473868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:54.474031 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.473938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:54.474093 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.474066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:54.474138 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.474096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:54.474190 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.474135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:54.474707 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.474679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-service-ca-bundle\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:54.476849 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.476823 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e1aabf-d4f3-4f85-83dc-72c8c5c61987-metrics-certs\") pod \"router-default-58f8796bd8-45bdh\" (UID: \"60e1aabf-d4f3-4f85-83dc-72c8c5c61987\") " pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:54.476939 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.476876 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c06db71-a2bf-4b00-82f2-3c1bc21555a0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cvm5x\" (UID: \"6c06db71-a2bf-4b00-82f2-3c1bc21555a0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:54.476998 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.476973 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5cfd20db-3fe7-47a1-bb11-8a8421cd83fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sppbq\" (UID: \"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:54.477556 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.477534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d785aaa-20ec-484a-a81e-48247bc83110-metrics-tls\") pod \"dns-default-2pxc6\" (UID: \"3d785aaa-20ec-484a-a81e-48247bc83110\") " pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:54.521355 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.521333 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6bpwz\"" Sep 16 07:34:54.529542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.529503 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:54.534095 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.534068 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-whsrb\"" Sep 16 07:34:54.542397 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.542380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" Sep 16 07:34:54.651790 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.651675 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4h2j4\"" Sep 16 07:34:54.660027 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.659929 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:54.672782 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.672741 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k8bpx\"" Sep 16 07:34:54.680425 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.680340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" Sep 16 07:34:54.688144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.688032 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2msf4\"" Sep 16 07:34:54.696579 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.696560 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:54.716444 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.713767 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-str4h\"" Sep 16 07:34:54.721932 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.721907 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" Sep 16 07:34:54.836401 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.836369 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79ffc6588-khnvp"] Sep 16 07:34:54.844222 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:54.844066 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfe1cf4_6dcd_424d_af1c_cb8d82c0b7fa.slice/crio-5261b388adf17598f6231eb4bfce74619a904e30c7155d24d1b3673c2e5a1f49 WatchSource:0}: Error finding container 5261b388adf17598f6231eb4bfce74619a904e30c7155d24d1b3673c2e5a1f49: Status 404 returned error can't find the container with id 5261b388adf17598f6231eb4bfce74619a904e30c7155d24d1b3673c2e5a1f49 Sep 16 07:34:54.858851 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.858826 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9"] Sep 16 07:34:54.877481 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.877257 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsx7w" event={"ID":"720450fe-3414-486b-b860-59fbdb3cfda5","Type":"ContainerStarted","Data":"dbd04a88e86a8b51930bf1d6962c6d17bf28e2f95e66b81a313d552760aca32a"} Sep 16 07:34:54.877481 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.877316 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsx7w" event={"ID":"720450fe-3414-486b-b860-59fbdb3cfda5","Type":"ContainerStarted","Data":"26ac7dbd21f3b1ae5b0f568ffa956b547c8c9e28798b8aaf54ae1c8b6d18c64c"} Sep 16 07:34:54.877481 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.877438 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:34:54.879547 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.879510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" event={"ID":"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa","Type":"ContainerStarted","Data":"5261b388adf17598f6231eb4bfce74619a904e30c7155d24d1b3673c2e5a1f49"} Sep 16 07:34:54.894290 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.894069 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xsx7w" podStartSLOduration=66.894049931 podStartE2EDuration="1m6.894049931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:54.892031416 +0000 UTC m=+67.811647363" watchObservedRunningTime="2026-09-16 07:34:54.894049931 +0000 UTC m=+67.813665879" Sep 16 07:34:54.965929 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.965895 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58f8796bd8-45bdh"] Sep 16 07:34:54.969633 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:54.969599 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e1aabf_d4f3_4f85_83dc_72c8c5c61987.slice/crio-d14568b811fbddc72245967190454ffe25550b41e9240224a59f93803d81f9fb WatchSource:0}: Error finding container d14568b811fbddc72245967190454ffe25550b41e9240224a59f93803d81f9fb: Status 404 returned error can't find the container with id d14568b811fbddc72245967190454ffe25550b41e9240224a59f93803d81f9fb Sep 16 07:34:54.994393 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:54.994318 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq"] Sep 16 07:34:55.001510 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:55.001478 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfd20db_3fe7_47a1_bb11_8a8421cd83fb.slice/crio-ccb788d32321738b628dbce35dd34df6564af65d71be225b45478df7f58ff9fe WatchSource:0}: Error finding container ccb788d32321738b628dbce35dd34df6564af65d71be225b45478df7f58ff9fe: Status 404 returned error can't find the container with id ccb788d32321738b628dbce35dd34df6564af65d71be225b45478df7f58ff9fe Sep 16 07:34:55.042716 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.042697 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x"] Sep 16 07:34:55.046332 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:55.046304 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c06db71_a2bf_4b00_82f2_3c1bc21555a0.slice/crio-0b2bcc055ea6ff86ddf7708e83ef6495b507f5e8e7ef5d2ae94b8d87b0e293ce WatchSource:0}: Error finding container 0b2bcc055ea6ff86ddf7708e83ef6495b507f5e8e7ef5d2ae94b8d87b0e293ce: Status 404 returned error can't find the container with id 0b2bcc055ea6ff86ddf7708e83ef6495b507f5e8e7ef5d2ae94b8d87b0e293ce Sep 16 07:34:55.047438 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.046924 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2pxc6"] Sep 16 07:34:55.049116 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:34:55.049092 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d785aaa_20ec_484a_a81e_48247bc83110.slice/crio-6475322239d593c4cbda6c9c850891ffad93574ad0d13a1bffc55692ea637dfb WatchSource:0}: Error finding container 6475322239d593c4cbda6c9c850891ffad93574ad0d13a1bffc55692ea637dfb: Status 404 returned error can't find the container with id 6475322239d593c4cbda6c9c850891ffad93574ad0d13a1bffc55692ea637dfb Sep 16 07:34:55.897979 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.897943 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58f8796bd8-45bdh" event={"ID":"60e1aabf-d4f3-4f85-83dc-72c8c5c61987","Type":"ContainerStarted","Data":"115c1456999bca6f546f28b92920ecf5c919edc327eac0ac16d4d8d7cc61d01f"} Sep 16 07:34:55.898446 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.898006 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58f8796bd8-45bdh" event={"ID":"60e1aabf-d4f3-4f85-83dc-72c8c5c61987","Type":"ContainerStarted","Data":"d14568b811fbddc72245967190454ffe25550b41e9240224a59f93803d81f9fb"} Sep 16 07:34:55.900743 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.900689 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" event={"ID":"8d486908-d637-4137-a9b1-f47a46e2aeac","Type":"ContainerStarted","Data":"e9fcbd81c673098a357b992ae8acd09c92c27a08d49372526059ee610d09f1b8"} Sep 16 07:34:55.904880 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.904420 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8qm8r" event={"ID":"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1","Type":"ContainerStarted","Data":"d447bd2521f87b6f8bac14bfa1dd647d178b5a30494ef91bf5cd189da7b2b005"} Sep 16 07:34:55.904880 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.904473 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8qm8r" event={"ID":"533591c0-bcd3-4f1f-9ba5-3eb2effa48a1","Type":"ContainerStarted","Data":"839de9f09695843c20a49e045d5c045c2873404a11f35df583e566aadb33bdc0"} Sep 16 07:34:55.907205 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.907144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" event={"ID":"6c06db71-a2bf-4b00-82f2-3c1bc21555a0","Type":"ContainerStarted","Data":"0b2bcc055ea6ff86ddf7708e83ef6495b507f5e8e7ef5d2ae94b8d87b0e293ce"} Sep 16 07:34:55.908784 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.908731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2pxc6" event={"ID":"3d785aaa-20ec-484a-a81e-48247bc83110","Type":"ContainerStarted","Data":"6475322239d593c4cbda6c9c850891ffad93574ad0d13a1bffc55692ea637dfb"} Sep 16 07:34:55.910891 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.910828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" event={"ID":"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb","Type":"ContainerStarted","Data":"ccb788d32321738b628dbce35dd34df6564af65d71be225b45478df7f58ff9fe"} Sep 16 07:34:55.914030 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.914000 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" event={"ID":"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa","Type":"ContainerStarted","Data":"e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4"} Sep 16 07:34:55.920253 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.919139 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-58f8796bd8-45bdh" podStartSLOduration=64.919122928 podStartE2EDuration="1m4.919122928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:55.917967425 +0000 UTC m=+68.837583372" watchObservedRunningTime="2026-09-16 07:34:55.919122928 +0000 UTC m=+68.838738877" Sep 16 07:34:55.938191 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.938141 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" podStartSLOduration=66.938124151 podStartE2EDuration="1m6.938124151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:55.937409745 +0000 UTC m=+68.857025690" watchObservedRunningTime="2026-09-16 07:34:55.938124151 +0000 UTC m=+68.857740098" Sep 16 07:34:55.952200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:55.952151 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8qm8r" podStartSLOduration=66.376757887 podStartE2EDuration="1m7.952137068s" totalImagesPullingTime="1.575379181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:55.951862713 +0000 UTC m=+68.871478662" watchObservedRunningTime="2026-09-16 07:34:55.952137068 +0000 UTC m=+68.871753015" Sep 16 07:34:56.660798 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:56.660704 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:56.663368 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:56.663347 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:56.920010 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:56.919955 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" event={"ID":"6c06db71-a2bf-4b00-82f2-3c1bc21555a0","Type":"ContainerStarted","Data":"4031631af6fec79538a50a6be98fd6a43fe33bf24592f7cbe5b63ab39ca3f058"} Sep 16 07:34:56.920555 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:56.920528 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:34:56.920780 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:56.920725 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:56.922197 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:56.922175 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-58f8796bd8-45bdh" Sep 16 07:34:56.943807 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:56.943754 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cvm5x" podStartSLOduration=50.389513641 podStartE2EDuration="51.943734836s" totalImagesPullingTime="1.554221195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:56.942278262 +0000 UTC m=+69.861894233" watchObservedRunningTime="2026-09-16 07:34:56.943734836 +0000 UTC m=+69.863350776" Sep 16 07:34:59.935964 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:59.935923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" event={"ID":"8d486908-d637-4137-a9b1-f47a46e2aeac","Type":"ContainerStarted","Data":"0bb05da4970a0ce9c8c71a41b5b607fc9a8dd3c32e5084c19e28cf5bbd3c80b5"} Sep 16 07:34:59.936424 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:59.936029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" event={"ID":"8d486908-d637-4137-a9b1-f47a46e2aeac","Type":"ContainerStarted","Data":"3885611194d485094f5be4bcf0bd074547f1ac2a78722cc28c15a07c1c71bc2f"} Sep 16 07:34:59.938038 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:59.938013 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2pxc6" event={"ID":"3d785aaa-20ec-484a-a81e-48247bc83110","Type":"ContainerStarted","Data":"ce73ab7a983c5481af44f092b3bf1bc7887512f2127ff3a3cd12136f1ec04106"} Sep 16 07:34:59.938178 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:59.938057 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2pxc6" event={"ID":"3d785aaa-20ec-484a-a81e-48247bc83110","Type":"ContainerStarted","Data":"0e4bd162a4fd0addef58329c2026152eea031163afc9363bb246fa31729e1cea"} Sep 16 07:34:59.938178 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:59.938088 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2pxc6" Sep 16 07:34:59.939905 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:59.939884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l2vjc" event={"ID":"b40f3f8d-04c4-4581-b929-fabed41b7079","Type":"ContainerStarted","Data":"645bebe5922df11f4c92ea2e327a6b9178e17ea4f1156ad4f2a90a13af84d7cf"} Sep 16 07:34:59.941624 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:59.941601 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" event={"ID":"5cfd20db-3fe7-47a1-bb11-8a8421cd83fb","Type":"ContainerStarted","Data":"e2f6afdc989f1eda141ca25774a56f742ed9af05e09e7b84ede7588d48947e14"} Sep 16 07:34:59.981042 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:34:59.980993 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-dwzp9" podStartSLOduration=64.290820847 podStartE2EDuration="1m8.980979378s" totalImagesPullingTime="4.690158531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:59.980202795 +0000 UTC m=+72.899818742" watchObservedRunningTime="2026-09-16 07:34:59.980979378 +0000 UTC m=+72.900595323" Sep 16 07:35:00.006713 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.006666 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sppbq" podStartSLOduration=64.421000081 podStartE2EDuration="1m9.006653713s" totalImagesPullingTime="4.585653632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:00.005483525 +0000 UTC m=+72.925099493" watchObservedRunningTime="2026-09-16 07:35:00.006653713 +0000 UTC m=+72.926269659" Sep 16 07:35:00.042566 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.042524 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l2vjc" podStartSLOduration=66.303814291 podStartE2EDuration="1m12.042511339s" totalImagesPullingTime="5.738697048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:00.041390326 +0000 UTC m=+72.961006273" watchObservedRunningTime="2026-09-16 07:35:00.042511339 +0000 UTC m=+72.962127284" Sep 16 07:35:00.063589 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.063550 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2pxc6" podStartSLOduration=33.524854014 podStartE2EDuration="38.063538244s" totalImagesPullingTime="4.53868423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:00.062867051 +0000 UTC m=+72.982483022" watchObservedRunningTime="2026-09-16 07:35:00.063538244 +0000 UTC m=+72.983154257" Sep 16 07:35:00.186122 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.186087 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-856c6f86fd-sbzj2"] Sep 16 07:35:00.192237 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.192203 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.195606 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.195584 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 07:35:00.195967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.195944 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 07:35:00.196300 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.196275 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 07:35:00.196405 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.196337 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 07:35:00.196685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.196663 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2pqpv\"" Sep 16 07:35:00.196771 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.196706 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 07:35:00.197048 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.197030 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 07:35:00.197360 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.197336 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 07:35:00.202725 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.202696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-856c6f86fd-sbzj2"] Sep 16 07:35:00.329134 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.329106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-serving-cert\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.329268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.329158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-console-config\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.329268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.329248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-oauth-config\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.329356 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.329278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-oauth-serving-cert\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.329356 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.329319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-service-ca\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.329356 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.329344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4h9r\" (UniqueName: \"kubernetes.io/projected/62844c41-3da9-4803-8758-945922651515-kube-api-access-k4h9r\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.430414 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.430386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-console-config\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.430542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.430432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-oauth-config\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.430542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.430459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-oauth-serving-cert\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.430542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.430497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-service-ca\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.430542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.430523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4h9r\" (UniqueName: \"kubernetes.io/projected/62844c41-3da9-4803-8758-945922651515-kube-api-access-k4h9r\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.430749 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.430571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-serving-cert\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.431260 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.431198 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-service-ca\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.431353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.431260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-oauth-serving-cert\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.431353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.431328 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-console-config\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.432984 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.432968 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-oauth-config\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.433046 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.433014 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-serving-cert\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.438892 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.438843 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4h9r\" (UniqueName: \"kubernetes.io/projected/62844c41-3da9-4803-8758-945922651515-kube-api-access-k4h9r\") pod \"console-856c6f86fd-sbzj2\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.503887 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.503851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:00.779697 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.779625 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-856c6f86fd-sbzj2"] Sep 16 07:35:00.782828 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:35:00.782797 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62844c41_3da9_4803_8758_945922651515.slice/crio-b644019767858b2279b15cb3d57fa0ba5e1351a48b9270cb60069fc635e3725c WatchSource:0}: Error finding container b644019767858b2279b15cb3d57fa0ba5e1351a48b9270cb60069fc635e3725c: Status 404 returned error can't find the container with id b644019767858b2279b15cb3d57fa0ba5e1351a48b9270cb60069fc635e3725c Sep 16 07:35:00.946633 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:00.946598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856c6f86fd-sbzj2" event={"ID":"62844c41-3da9-4803-8758-945922651515","Type":"ContainerStarted","Data":"b644019767858b2279b15cb3d57fa0ba5e1351a48b9270cb60069fc635e3725c"} Sep 16 07:35:04.964962 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:04.964881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856c6f86fd-sbzj2" event={"ID":"62844c41-3da9-4803-8758-945922651515","Type":"ContainerStarted","Data":"05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e"} Sep 16 07:35:04.981961 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:04.981907 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-856c6f86fd-sbzj2" podStartSLOduration=1.08441921 podStartE2EDuration="4.981889077s" totalImagesPullingTime="3.897469867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:04.98177676 +0000 UTC m=+77.901392707" watchObservedRunningTime="2026-09-16 07:35:04.981889077 +0000 UTC m=+77.901505027" Sep 16 07:35:09.948354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:09.948318 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2pxc6" Sep 16 07:35:10.504224 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.504174 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:10.504362 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.504260 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:10.509182 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.509154 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:10.643959 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.643928 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m"] Sep 16 07:35:10.646647 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.646626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.649814 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.649791 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 16 07:35:10.649917 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.649823 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 16 07:35:10.649917 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.649893 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x4trf\"" Sep 16 07:35:10.650033 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.649944 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 16 07:35:10.650150 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.650136 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 07:35:10.662678 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.662649 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m"] Sep 16 07:35:10.663755 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.663735 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8prhl"] Sep 16 07:35:10.666004 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.665986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.668259 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.668239 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 07:35:10.668446 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.668417 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 07:35:10.668554 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.668540 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 07:35:10.668620 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.668578 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kqtql\"" Sep 16 07:35:10.810350 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc9t2\" (UniqueName: \"kubernetes.io/projected/3a4c6bf0-be34-4bbc-8473-246793553bda-kube-api-access-hc9t2\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.810350 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.810350 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810335 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a4c6bf0-be34-4bbc-8473-246793553bda-root\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.810535 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-textfile\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.810535 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810395 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.810535 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810426 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a4c6bf0-be34-4bbc-8473-246793553bda-metrics-client-ca\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.810535 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.810535 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810483 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-tls\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.810535 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.810712 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810556 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.810712 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.810712 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqqs9\" (UniqueName: \"kubernetes.io/projected/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-api-access-kqqs9\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.810712 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-wtmp\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.810712 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a4c6bf0-be34-4bbc-8473-246793553bda-sys\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.810712 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.810680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-accelerators-collector-config\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.911915 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.911889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.912085 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.911924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-tls\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.912085 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.911947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.912085 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:35:10.912068 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 07:35:10.912283 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:35:10.912140 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-tls podName:3a4c6bf0-be34-4bbc-8473-246793553bda nodeName:}" failed. No retries permitted until 2026-09-16 07:35:11.412118322 +0000 UTC m=+84.331734257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-tls") pod "node-exporter-8prhl" (UID: "3a4c6bf0-be34-4bbc-8473-246793553bda") : secret "node-exporter-tls" not found Sep 16 07:35:10.912283 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.912283 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.912410 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqqs9\" (UniqueName: \"kubernetes.io/projected/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-api-access-kqqs9\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.912410 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.912410 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-wtmp\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.912529 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a4c6bf0-be34-4bbc-8473-246793553bda-sys\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.912529 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-accelerators-collector-config\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.912529 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-wtmp\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.912529 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc9t2\" (UniqueName: \"kubernetes.io/projected/3a4c6bf0-be34-4bbc-8473-246793553bda-kube-api-access-hc9t2\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.912529 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a4c6bf0-be34-4bbc-8473-246793553bda-sys\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.912769 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912560 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.912769 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a4c6bf0-be34-4bbc-8473-246793553bda-root\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.912769 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-textfile\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.912769 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912654 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.912769 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a4c6bf0-be34-4bbc-8473-246793553bda-root\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.912769 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.913045 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.912786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a4c6bf0-be34-4bbc-8473-246793553bda-metrics-client-ca\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.913112 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.913041 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-textfile\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.913197 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.913173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-accelerators-collector-config\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.913300 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.913283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.913354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.913334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a4c6bf0-be34-4bbc-8473-246793553bda-metrics-client-ca\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.914890 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.914865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.914995 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.914964 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.915050 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.915033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.921898 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.921871 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc9t2\" (UniqueName: \"kubernetes.io/projected/3a4c6bf0-be34-4bbc-8473-246793553bda-kube-api-access-hc9t2\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:10.922060 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.922042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqqs9\" (UniqueName: \"kubernetes.io/projected/d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5-kube-api-access-kqqs9\") pod \"kube-state-metrics-59fbdd77d-fhw4m\" (UID: \"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.925018 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.924994 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56b8bbf5dd-7h9sf"] Sep 16 07:35:10.927597 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.927578 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:10.934125 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.934105 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 07:35:10.938938 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.938915 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b8bbf5dd-7h9sf"] Sep 16 07:35:10.956024 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.956007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" Sep 16 07:35:10.994490 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:10.994464 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:11.114359 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.114272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5nb\" (UniqueName: \"kubernetes.io/projected/f22c01a0-e8d3-477a-9655-584f7d204eeb-kube-api-access-nt5nb\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.114359 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.114315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-oauth-serving-cert\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.114541 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.114446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-oauth-config\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.114594 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.114535 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-trusted-ca-bundle\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.114594 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.114565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-config\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.114672 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.114644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-serving-cert\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.114725 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.114711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-service-ca\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.215716 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.215685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5nb\" (UniqueName: \"kubernetes.io/projected/f22c01a0-e8d3-477a-9655-584f7d204eeb-kube-api-access-nt5nb\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.215716 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.215716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-oauth-serving-cert\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.215906 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.215752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-oauth-config\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.215906 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.215777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-trusted-ca-bundle\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.215906 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.215801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-config\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.215906 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.215827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-serving-cert\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.215906 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.215868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-service-ca\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.216528 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.216500 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-oauth-serving-cert\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.216661 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.216535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-service-ca\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.216661 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.216626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-config\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.216779 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.216753 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-trusted-ca-bundle\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.218312 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.218290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-oauth-config\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.218312 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.218307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-serving-cert\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.223468 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.223422 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5nb\" (UniqueName: \"kubernetes.io/projected/f22c01a0-e8d3-477a-9655-584f7d204eeb-kube-api-access-nt5nb\") pod \"console-56b8bbf5dd-7h9sf\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.225766 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.225749 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m"] Sep 16 07:35:11.227986 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:35:11.227967 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0351e2b_2cff_4db7_9e0f_b5cbae9f4bb5.slice/crio-c8252e756ea3db752ac8c67290cc45aac1a9853f2437afce71a16a6dfe91ae3c WatchSource:0}: Error finding container c8252e756ea3db752ac8c67290cc45aac1a9853f2437afce71a16a6dfe91ae3c: Status 404 returned error can't find the container with id c8252e756ea3db752ac8c67290cc45aac1a9853f2437afce71a16a6dfe91ae3c Sep 16 07:35:11.238703 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.238685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:11.418117 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.418087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-tls\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:11.420666 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.420648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a4c6bf0-be34-4bbc-8473-246793553bda-node-exporter-tls\") pod \"node-exporter-8prhl\" (UID: \"3a4c6bf0-be34-4bbc-8473-246793553bda\") " pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:11.504001 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.503929 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b8bbf5dd-7h9sf"] Sep 16 07:35:11.505960 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:35:11.505933 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22c01a0_e8d3_477a_9655_584f7d204eeb.slice/crio-0609885610cc556459789785046fa0c781b1800b2057598cb345de83a6c28b82 WatchSource:0}: Error finding container 0609885610cc556459789785046fa0c781b1800b2057598cb345de83a6c28b82: Status 404 returned error can't find the container with id 0609885610cc556459789785046fa0c781b1800b2057598cb345de83a6c28b82 Sep 16 07:35:11.574711 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.574693 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8prhl" Sep 16 07:35:11.584164 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:35:11.584131 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4c6bf0_be34_4bbc_8473_246793553bda.slice/crio-b2ff2192b78aecfdeef754a7c4d5382f13ff8aa908e53c61b2d390b638237589 WatchSource:0}: Error finding container b2ff2192b78aecfdeef754a7c4d5382f13ff8aa908e53c61b2d390b638237589: Status 404 returned error can't find the container with id b2ff2192b78aecfdeef754a7c4d5382f13ff8aa908e53c61b2d390b638237589 Sep 16 07:35:11.996296 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.996248 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8prhl" event={"ID":"3a4c6bf0-be34-4bbc-8473-246793553bda","Type":"ContainerStarted","Data":"b2ff2192b78aecfdeef754a7c4d5382f13ff8aa908e53c61b2d390b638237589"} Sep 16 07:35:11.998733 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.998702 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b8bbf5dd-7h9sf" event={"ID":"f22c01a0-e8d3-477a-9655-584f7d204eeb","Type":"ContainerStarted","Data":"ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273"} Sep 16 07:35:11.998851 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:11.998750 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b8bbf5dd-7h9sf" event={"ID":"f22c01a0-e8d3-477a-9655-584f7d204eeb","Type":"ContainerStarted","Data":"0609885610cc556459789785046fa0c781b1800b2057598cb345de83a6c28b82"} Sep 16 07:35:12.000434 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.000400 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" event={"ID":"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5","Type":"ContainerStarted","Data":"c8252e756ea3db752ac8c67290cc45aac1a9853f2437afce71a16a6dfe91ae3c"} Sep 16 07:35:12.017802 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.017749 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56b8bbf5dd-7h9sf" podStartSLOduration=2.017729221 podStartE2EDuration="2.017729221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:12.01525628 +0000 UTC m=+84.934872227" watchObservedRunningTime="2026-09-16 07:35:12.017729221 +0000 UTC m=+84.937345168" Sep 16 07:35:12.627091 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.627056 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-54889c48b5-pbxts"] Sep 16 07:35:12.631129 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.631101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.633805 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.633781 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-64gqg\"" Sep 16 07:35:12.633805 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.633781 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5eht0ohlu2ijo\"" Sep 16 07:35:12.633997 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.633929 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 07:35:12.634083 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.634019 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 07:35:12.634486 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.634463 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 07:35:12.634988 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.634968 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 07:35:12.636131 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.635552 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 07:35:12.643381 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.643296 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54889c48b5-pbxts"] Sep 16 07:35:12.728400 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.728367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-grpc-tls\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.728499 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.728447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fc426e4-6f37-4064-884c-0b6e92572f18-metrics-client-ca\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.728499 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.728478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.728607 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.728512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.728607 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.728544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-tls\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.728765 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.728679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.728765 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.728721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rvw\" (UniqueName: \"kubernetes.io/projected/9fc426e4-6f37-4064-884c-0b6e92572f18-kube-api-access-p8rvw\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.728855 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.728789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.829699 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.829665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.829928 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.829713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rvw\" (UniqueName: \"kubernetes.io/projected/9fc426e4-6f37-4064-884c-0b6e92572f18-kube-api-access-p8rvw\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.829928 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.829765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.829928 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.829799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-grpc-tls\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.829928 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.829857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fc426e4-6f37-4064-884c-0b6e92572f18-metrics-client-ca\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.829928 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.829884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.829928 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.829911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.830278 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.829942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-tls\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.830908 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.830840 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fc426e4-6f37-4064-884c-0b6e92572f18-metrics-client-ca\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.833652 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.833597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.833652 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.833643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.834303 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.834279 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.834697 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.834654 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.835320 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.835135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-thanos-querier-tls\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.835320 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.835290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fc426e4-6f37-4064-884c-0b6e92572f18-secret-grpc-tls\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.838964 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.838936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rvw\" (UniqueName: \"kubernetes.io/projected/9fc426e4-6f37-4064-884c-0b6e92572f18-kube-api-access-p8rvw\") pod \"thanos-querier-54889c48b5-pbxts\" (UID: \"9fc426e4-6f37-4064-884c-0b6e92572f18\") " pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:12.945913 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:12.945874 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:13.010485 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:13.009788 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a4c6bf0-be34-4bbc-8473-246793553bda" containerID="a04297bf2bd3ff806e1bfc041d914372648c2aaf572510f50a7a98712d706961" exitCode=0 Sep 16 07:35:13.010485 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:13.009903 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8prhl" event={"ID":"3a4c6bf0-be34-4bbc-8473-246793553bda","Type":"ContainerDied","Data":"a04297bf2bd3ff806e1bfc041d914372648c2aaf572510f50a7a98712d706961"} Sep 16 07:35:13.014350 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:13.014158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" event={"ID":"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5","Type":"ContainerStarted","Data":"29dbae555897661947d805acc788a9fef3af65e3c151410e1460785bcf6cdf3e"} Sep 16 07:35:13.014350 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:13.014226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" event={"ID":"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5","Type":"ContainerStarted","Data":"397e4ed5f4203681863291baabf91971f00d3c1f08aa165283123260dac8b130"} Sep 16 07:35:13.014350 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:13.014256 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" event={"ID":"d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5","Type":"ContainerStarted","Data":"2cd30005dafbd9734298a5450c9365242a0569e8bb05d7b4be9df498f24ec05a"} Sep 16 07:35:13.049555 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:13.049513 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fhw4m" podStartSLOduration=1.5486206679999999 podStartE2EDuration="3.049497361s" totalImagesPullingTime="1.500876693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:13.048008057 +0000 UTC m=+85.967624006" watchObservedRunningTime="2026-09-16 07:35:13.049497361 +0000 UTC m=+85.969113308" Sep 16 07:35:13.223482 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:13.223433 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54889c48b5-pbxts"] Sep 16 07:35:13.225482 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:35:13.225452 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc426e4_6f37_4064_884c_0b6e92572f18.slice/crio-731785251b78cedecd8f30863dbc131663e82492640893883dab393a0c26b094 WatchSource:0}: Error finding container 731785251b78cedecd8f30863dbc131663e82492640893883dab393a0c26b094: Status 404 returned error can't find the container with id 731785251b78cedecd8f30863dbc131663e82492640893883dab393a0c26b094 Sep 16 07:35:14.018587 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.018549 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" event={"ID":"9fc426e4-6f37-4064-884c-0b6e92572f18","Type":"ContainerStarted","Data":"731785251b78cedecd8f30863dbc131663e82492640893883dab393a0c26b094"} Sep 16 07:35:14.020778 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.020752 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8prhl" event={"ID":"3a4c6bf0-be34-4bbc-8473-246793553bda","Type":"ContainerStarted","Data":"79af7a431db5edaa5dac922bf675d1d42468cf496b3651eb1545882290c82f8d"} Sep 16 07:35:14.020920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.020797 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8prhl" event={"ID":"3a4c6bf0-be34-4bbc-8473-246793553bda","Type":"ContainerStarted","Data":"4280edd8e31bc9e5b60469891f79825b5f9d0929d86812a35e1803698febdbc2"} Sep 16 07:35:14.039322 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.039279 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8prhl" podStartSLOduration=2.810097466 podStartE2EDuration="4.03926487s" totalImagesPullingTime="1.145762333s" totalInitContainerRuntime="83.405071ms" isStatefulPod=true podCreationTimestamp="2026-09-16 07:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:14.038847661 +0000 UTC m=+86.958463629" watchObservedRunningTime="2026-09-16 07:35:14.03926487 +0000 UTC m=+86.958880816" Sep 16 07:35:14.948143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.948102 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-584ff4576-m57c9"] Sep 16 07:35:14.950423 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.950403 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:14.954232 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.954193 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c14uanbb676f9\"" Sep 16 07:35:14.954332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.954292 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 16 07:35:14.954332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.954201 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 07:35:14.954458 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.954330 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 16 07:35:14.954458 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.954343 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 16 07:35:14.954458 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.954398 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6nmfp\"" Sep 16 07:35:14.961289 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:14.961267 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-584ff4576-m57c9"] Sep 16 07:35:15.048862 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.048828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/40595f6f-bce1-42a4-ad70-d5591b98bd9e-audit-log\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.049345 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.048868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40595f6f-bce1-42a4-ad70-d5591b98bd9e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.049345 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.048895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96dr9\" (UniqueName: \"kubernetes.io/projected/40595f6f-bce1-42a4-ad70-d5591b98bd9e-kube-api-access-96dr9\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.049345 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.049001 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/40595f6f-bce1-42a4-ad70-d5591b98bd9e-client-ca-bundle\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.049345 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.049045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/40595f6f-bce1-42a4-ad70-d5591b98bd9e-secret-metrics-server-client-certs\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.049345 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.049079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/40595f6f-bce1-42a4-ad70-d5591b98bd9e-secret-metrics-server-tls\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.049345 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.049123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/40595f6f-bce1-42a4-ad70-d5591b98bd9e-metrics-server-audit-profiles\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.149632 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.149602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/40595f6f-bce1-42a4-ad70-d5591b98bd9e-audit-log\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.149785 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.149638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40595f6f-bce1-42a4-ad70-d5591b98bd9e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.149785 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.149663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96dr9\" (UniqueName: \"kubernetes.io/projected/40595f6f-bce1-42a4-ad70-d5591b98bd9e-kube-api-access-96dr9\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.149785 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.149743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/40595f6f-bce1-42a4-ad70-d5591b98bd9e-client-ca-bundle\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.149958 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.149939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/40595f6f-bce1-42a4-ad70-d5591b98bd9e-secret-metrics-server-client-certs\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.150028 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.149979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/40595f6f-bce1-42a4-ad70-d5591b98bd9e-secret-metrics-server-tls\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.150028 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.150015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/40595f6f-bce1-42a4-ad70-d5591b98bd9e-metrics-server-audit-profiles\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.150263 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.150184 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/40595f6f-bce1-42a4-ad70-d5591b98bd9e-audit-log\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.150482 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.150440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40595f6f-bce1-42a4-ad70-d5591b98bd9e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.151105 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.151085 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/40595f6f-bce1-42a4-ad70-d5591b98bd9e-metrics-server-audit-profiles\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.153021 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.152985 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/40595f6f-bce1-42a4-ad70-d5591b98bd9e-secret-metrics-server-client-certs\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.153110 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.152995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/40595f6f-bce1-42a4-ad70-d5591b98bd9e-secret-metrics-server-tls\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.153551 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.153527 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/40595f6f-bce1-42a4-ad70-d5591b98bd9e-client-ca-bundle\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.158103 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.158083 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96dr9\" (UniqueName: \"kubernetes.io/projected/40595f6f-bce1-42a4-ad70-d5591b98bd9e-kube-api-access-96dr9\") pod \"metrics-server-584ff4576-m57c9\" (UID: \"40595f6f-bce1-42a4-ad70-d5591b98bd9e\") " pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.260347 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.260271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:15.741949 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:15.741926 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-584ff4576-m57c9"] Sep 16 07:35:15.743531 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:35:15.743505 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40595f6f_bce1_42a4_ad70_d5591b98bd9e.slice/crio-3248c39cc9b222f9ae28d8c23a227c4c8f11370797f73728f0ad05322bb8d91a WatchSource:0}: Error finding container 3248c39cc9b222f9ae28d8c23a227c4c8f11370797f73728f0ad05322bb8d91a: Status 404 returned error can't find the container with id 3248c39cc9b222f9ae28d8c23a227c4c8f11370797f73728f0ad05322bb8d91a Sep 16 07:35:16.031724 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:16.031691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-584ff4576-m57c9" event={"ID":"40595f6f-bce1-42a4-ad70-d5591b98bd9e","Type":"ContainerStarted","Data":"3248c39cc9b222f9ae28d8c23a227c4c8f11370797f73728f0ad05322bb8d91a"} Sep 16 07:35:16.033643 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:16.033619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" event={"ID":"9fc426e4-6f37-4064-884c-0b6e92572f18","Type":"ContainerStarted","Data":"3caf11a47e5caefac905e189e0cd0ccf52f8c5743b1b3b45d6ca66517d5d1380"} Sep 16 07:35:16.033764 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:16.033675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" event={"ID":"9fc426e4-6f37-4064-884c-0b6e92572f18","Type":"ContainerStarted","Data":"bca861dd3ce975706a9b9f4aa5e4cbf5c4451652e3d962a9509f75ec1eafd3db"} Sep 16 07:35:16.033764 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:16.033704 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" event={"ID":"9fc426e4-6f37-4064-884c-0b6e92572f18","Type":"ContainerStarted","Data":"71a60dfd6cac4c92b021e5b6772d859dd6f0de157ec42838342cf81bde7a0480"} Sep 16 07:35:17.042268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:17.042203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" event={"ID":"9fc426e4-6f37-4064-884c-0b6e92572f18","Type":"ContainerStarted","Data":"f96ec860c268c32fc2ef7efd87fad35ef4e4b5988c6cb8619afb031fa91d27f0"} Sep 16 07:35:17.042704 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:17.042314 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" event={"ID":"9fc426e4-6f37-4064-884c-0b6e92572f18","Type":"ContainerStarted","Data":"e28476f866b54bb20fa7427d338da4c7a08b5f29a0ef1657428ff28736800889"} Sep 16 07:35:17.042704 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:17.042353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" event={"ID":"9fc426e4-6f37-4064-884c-0b6e92572f18","Type":"ContainerStarted","Data":"ab51402c5c63387faed7ece9d880c7057820b2525deb3063b2e4247a3b35a943"} Sep 16 07:35:17.042704 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:17.042496 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:17.066449 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:17.066388 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" podStartSLOduration=1.9173201450000001 podStartE2EDuration="5.066373033s" totalImagesPullingTime="3.149052888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:17.063782925 +0000 UTC m=+89.983398872" watchObservedRunningTime="2026-09-16 07:35:17.066373033 +0000 UTC m=+89.985988977" Sep 16 07:35:17.928687 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:17.928657 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:35:18.048723 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:18.048693 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-584ff4576-m57c9" event={"ID":"40595f6f-bce1-42a4-ad70-d5591b98bd9e","Type":"ContainerStarted","Data":"64c5e2c5849c1a74950632be6938a870bdac281a85f7db56c7c944b9fd8f3764"} Sep 16 07:35:18.068166 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:18.068118 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-584ff4576-m57c9" podStartSLOduration=2.502271325 podStartE2EDuration="4.068102998s" totalImagesPullingTime="1.565831673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:18.066867263 +0000 UTC m=+90.986483209" watchObservedRunningTime="2026-09-16 07:35:18.068102998 +0000 UTC m=+90.987718942" Sep 16 07:35:19.306652 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.306612 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79ffc6588-khnvp"] Sep 16 07:35:19.536290 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.536258 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b8bbf5dd-7h9sf"] Sep 16 07:35:19.563132 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.563062 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c5577f8c9-fvpnq"] Sep 16 07:35:19.565658 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.565636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.579421 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.579396 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5577f8c9-fvpnq"] Sep 16 07:35:19.690165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.689081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-oauth-serving-cert\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.690165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.689131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-config\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.690165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.689187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-serving-cert\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.690165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.689229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-trusted-ca-bundle\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.690165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.689254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92s77\" (UniqueName: \"kubernetes.io/projected/30ab6452-4aa4-45ed-b22d-9ab75df94a99-kube-api-access-92s77\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.690165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.689300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-oauth-config\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.690165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.689322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-service-ca\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.789990 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.789962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-serving-cert\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.790119 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.789994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-trusted-ca-bundle\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.790119 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.790012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92s77\" (UniqueName: \"kubernetes.io/projected/30ab6452-4aa4-45ed-b22d-9ab75df94a99-kube-api-access-92s77\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.790269 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.790146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-oauth-config\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.790269 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.790184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-service-ca\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.790369 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.790276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-oauth-serving-cert\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.790369 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.790308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-config\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.790972 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.790943 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-config\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.791088 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.791000 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-trusted-ca-bundle\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.791088 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.791044 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-oauth-serving-cert\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.791088 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.791073 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-service-ca\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.792577 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.792548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-serving-cert\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.792666 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.792627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-oauth-config\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.797651 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.797634 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92s77\" (UniqueName: \"kubernetes.io/projected/30ab6452-4aa4-45ed-b22d-9ab75df94a99-kube-api-access-92s77\") pod \"console-c5577f8c9-fvpnq\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:19.874446 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:19.874392 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:20.141053 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:20.141025 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5577f8c9-fvpnq"] Sep 16 07:35:20.143496 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:35:20.143469 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ab6452_4aa4_45ed_b22d_9ab75df94a99.slice/crio-18481b8a81d19bb24d896385d6ce6fda081e1a3be3055133120351c2d3affb24 WatchSource:0}: Error finding container 18481b8a81d19bb24d896385d6ce6fda081e1a3be3055133120351c2d3affb24: Status 404 returned error can't find the container with id 18481b8a81d19bb24d896385d6ce6fda081e1a3be3055133120351c2d3affb24 Sep 16 07:35:21.062628 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:21.062588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5577f8c9-fvpnq" event={"ID":"30ab6452-4aa4-45ed-b22d-9ab75df94a99","Type":"ContainerStarted","Data":"d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c"} Sep 16 07:35:21.062994 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:21.062638 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5577f8c9-fvpnq" event={"ID":"30ab6452-4aa4-45ed-b22d-9ab75df94a99","Type":"ContainerStarted","Data":"18481b8a81d19bb24d896385d6ce6fda081e1a3be3055133120351c2d3affb24"} Sep 16 07:35:21.089963 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:21.089917 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c5577f8c9-fvpnq" podStartSLOduration=2.089902808 podStartE2EDuration="2.089902808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:21.088092233 +0000 UTC m=+94.007708181" watchObservedRunningTime="2026-09-16 07:35:21.089902808 +0000 UTC m=+94.009518754" Sep 16 07:35:21.238781 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:21.238756 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:23.055875 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:23.055848 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-54889c48b5-pbxts" Sep 16 07:35:25.916472 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:25.916444 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xsx7w" Sep 16 07:35:29.875596 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:29.875561 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:29.875944 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:29.875619 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:29.880411 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:29.880387 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:30.108083 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:30.108053 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:35:30.161532 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:30.161503 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-856c6f86fd-sbzj2"] Sep 16 07:35:35.260724 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:35.260695 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:35.261167 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:35.260748 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:44.327732 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.327671 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" podUID="dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" containerName="registry" containerID="cri-o://e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4" gracePeriod=30 Sep 16 07:35:44.555564 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.555524 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56b8bbf5dd-7h9sf" podUID="f22c01a0-e8d3-477a-9655-584f7d204eeb" containerName="console" containerID="cri-o://ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273" gracePeriod=15 Sep 16 07:35:44.572315 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.572293 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:35:44.673112 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.673091 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-installation-pull-secrets\") pod \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " Sep 16 07:35:44.673173 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.673127 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-certificates\") pod \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " Sep 16 07:35:44.673301 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.673181 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-ca-trust-extracted\") pod \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " Sep 16 07:35:44.673301 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.673278 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-image-registry-private-configuration\") pod \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " Sep 16 07:35:44.673425 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.673325 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-bound-sa-token\") pod \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " Sep 16 07:35:44.673425 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.673356 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-trusted-ca\") pod \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " Sep 16 07:35:44.673425 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.673390 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") pod \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " Sep 16 07:35:44.673578 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.673442 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtvpg\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-kube-api-access-jtvpg\") pod \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\" (UID: \"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa\") " Sep 16 07:35:44.673857 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.673820 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:44.674006 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.673975 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:44.676127 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.676094 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:35:44.676308 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.676168 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-kube-api-access-jtvpg" (OuterVolumeSpecName: "kube-api-access-jtvpg") pod "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa"). InnerVolumeSpecName "kube-api-access-jtvpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:35:44.676308 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.676237 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:35:44.676444 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.676316 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:35:44.676444 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.676367 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:35:44.681964 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.681939 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" (UID: "dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:35:44.774922 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.774893 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-installation-pull-secrets\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.774922 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.774919 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-certificates\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.775092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.774928 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-ca-trust-extracted\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.775092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.774938 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-image-registry-private-configuration\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.775092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.774948 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-bound-sa-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.775092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.774957 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-trusted-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.775092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.774965 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-registry-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.775092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.774975 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtvpg\" (UniqueName: \"kubernetes.io/projected/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa-kube-api-access-jtvpg\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.788068 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.788047 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:44.875967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.875938 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-service-ca\") pod \"f22c01a0-e8d3-477a-9655-584f7d204eeb\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " Sep 16 07:35:44.876101 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.875994 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-config\") pod \"f22c01a0-e8d3-477a-9655-584f7d204eeb\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " Sep 16 07:35:44.876101 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.876034 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-oauth-config\") pod \"f22c01a0-e8d3-477a-9655-584f7d204eeb\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " Sep 16 07:35:44.876101 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.876081 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt5nb\" (UniqueName: \"kubernetes.io/projected/f22c01a0-e8d3-477a-9655-584f7d204eeb-kube-api-access-nt5nb\") pod \"f22c01a0-e8d3-477a-9655-584f7d204eeb\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " Sep 16 07:35:44.876285 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.876116 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-oauth-serving-cert\") pod \"f22c01a0-e8d3-477a-9655-584f7d204eeb\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " Sep 16 07:35:44.876285 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.876147 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-trusted-ca-bundle\") pod \"f22c01a0-e8d3-477a-9655-584f7d204eeb\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " Sep 16 07:35:44.876285 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.876190 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-serving-cert\") pod \"f22c01a0-e8d3-477a-9655-584f7d204eeb\" (UID: \"f22c01a0-e8d3-477a-9655-584f7d204eeb\") " Sep 16 07:35:44.876418 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.876326 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-service-ca" (OuterVolumeSpecName: "service-ca") pod "f22c01a0-e8d3-477a-9655-584f7d204eeb" (UID: "f22c01a0-e8d3-477a-9655-584f7d204eeb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:44.876512 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.876370 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-config" (OuterVolumeSpecName: "console-config") pod "f22c01a0-e8d3-477a-9655-584f7d204eeb" (UID: "f22c01a0-e8d3-477a-9655-584f7d204eeb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:44.876512 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.876499 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.876663 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.876541 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f22c01a0-e8d3-477a-9655-584f7d204eeb" (UID: "f22c01a0-e8d3-477a-9655-584f7d204eeb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:44.876663 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.876571 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f22c01a0-e8d3-477a-9655-584f7d204eeb" (UID: "f22c01a0-e8d3-477a-9655-584f7d204eeb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:44.878392 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.878364 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f22c01a0-e8d3-477a-9655-584f7d204eeb" (UID: "f22c01a0-e8d3-477a-9655-584f7d204eeb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:35:44.878468 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.878404 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f22c01a0-e8d3-477a-9655-584f7d204eeb" (UID: "f22c01a0-e8d3-477a-9655-584f7d204eeb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:35:44.878468 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.878420 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22c01a0-e8d3-477a-9655-584f7d204eeb-kube-api-access-nt5nb" (OuterVolumeSpecName: "kube-api-access-nt5nb") pod "f22c01a0-e8d3-477a-9655-584f7d204eeb" (UID: "f22c01a0-e8d3-477a-9655-584f7d204eeb"). InnerVolumeSpecName "kube-api-access-nt5nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:35:44.977077 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.977024 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt5nb\" (UniqueName: \"kubernetes.io/projected/f22c01a0-e8d3-477a-9655-584f7d204eeb-kube-api-access-nt5nb\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.977077 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.977044 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-oauth-serving-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.977077 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.977054 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-trusted-ca-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.977077 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.977063 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-serving-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.977077 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.977071 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:44.977077 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:44.977080 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f22c01a0-e8d3-477a-9655-584f7d204eeb-console-oauth-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:45.169083 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.169049 2580 generic.go:356] "Generic (PLEG): container finished" podID="dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" containerID="e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4" exitCode=0 Sep 16 07:35:45.169240 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.169097 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" Sep 16 07:35:45.169240 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.169138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" event={"ID":"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa","Type":"ContainerDied","Data":"e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4"} Sep 16 07:35:45.169240 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.169190 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79ffc6588-khnvp" event={"ID":"dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa","Type":"ContainerDied","Data":"5261b388adf17598f6231eb4bfce74619a904e30c7155d24d1b3673c2e5a1f49"} Sep 16 07:35:45.169240 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.169239 2580 scope.go:117] "RemoveContainer" containerID="e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4" Sep 16 07:35:45.170658 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.170638 2580 generic.go:356] "Generic (PLEG): container finished" podID="f22c01a0-e8d3-477a-9655-584f7d204eeb" containerID="ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273" exitCode=0 Sep 16 07:35:45.170762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.170684 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b8bbf5dd-7h9sf" Sep 16 07:35:45.170762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.170705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b8bbf5dd-7h9sf" event={"ID":"f22c01a0-e8d3-477a-9655-584f7d204eeb","Type":"ContainerDied","Data":"ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273"} Sep 16 07:35:45.170762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.170747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b8bbf5dd-7h9sf" event={"ID":"f22c01a0-e8d3-477a-9655-584f7d204eeb","Type":"ContainerDied","Data":"0609885610cc556459789785046fa0c781b1800b2057598cb345de83a6c28b82"} Sep 16 07:35:45.177286 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.177268 2580 scope.go:117] "RemoveContainer" containerID="e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4" Sep 16 07:35:45.177577 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:35:45.177551 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4\": container with ID starting with e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4 not found: ID does not exist" containerID="e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4" Sep 16 07:35:45.177635 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.177587 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4"} err="failed to get container status \"e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4\": rpc error: code = NotFound desc = could not find container \"e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4\": container with ID starting with e540535324f64258b970c0ba9a7c1a635f778ceaca8ec3473423f0fffad4d1e4 not found: ID does not exist" Sep 16 07:35:45.177635 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.177624 2580 scope.go:117] "RemoveContainer" containerID="ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273" Sep 16 07:35:45.185663 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.185648 2580 scope.go:117] "RemoveContainer" containerID="ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273" Sep 16 07:35:45.185938 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:35:45.185919 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273\": container with ID starting with ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273 not found: ID does not exist" containerID="ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273" Sep 16 07:35:45.186018 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.185943 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273"} err="failed to get container status \"ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273\": rpc error: code = NotFound desc = could not find container \"ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273\": container with ID starting with ce0a2f17b20f3cb8d51025c62935f2242bf8f959f49f6bd82a7afc3596aba273 not found: ID does not exist" Sep 16 07:35:45.194569 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.194551 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79ffc6588-khnvp"] Sep 16 07:35:45.198124 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.198098 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79ffc6588-khnvp"] Sep 16 07:35:45.207750 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.207730 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b8bbf5dd-7h9sf"] Sep 16 07:35:45.211276 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.211247 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56b8bbf5dd-7h9sf"] Sep 16 07:35:45.558927 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.558898 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" path="/var/lib/kubelet/pods/dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa/volumes" Sep 16 07:35:45.559313 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:45.559298 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f22c01a0-e8d3-477a-9655-584f7d204eeb" path="/var/lib/kubelet/pods/f22c01a0-e8d3-477a-9655-584f7d204eeb/volumes" Sep 16 07:35:52.202858 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:52.202819 2580 generic.go:356] "Generic (PLEG): container finished" podID="f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b" containerID="e59548f6a9265209b1c1762de9fb7d3da0b885a1311488418586dba4869e4717" exitCode=0 Sep 16 07:35:52.203356 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:52.202886 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" event={"ID":"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b","Type":"ContainerDied","Data":"e59548f6a9265209b1c1762de9fb7d3da0b885a1311488418586dba4869e4717"} Sep 16 07:35:52.203356 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:52.203348 2580 scope.go:117] "RemoveContainer" containerID="e59548f6a9265209b1c1762de9fb7d3da0b885a1311488418586dba4869e4717" Sep 16 07:35:53.208812 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:53.208772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-btmxg" event={"ID":"f8b1b2ad-ea4e-48bb-a17d-10a2df9cf74b","Type":"ContainerStarted","Data":"edf9b56869d7f8df92b5c514273d9e0b089f16953172d1494205049b467d9a4f"} Sep 16 07:35:55.181734 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.181690 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-856c6f86fd-sbzj2" podUID="62844c41-3da9-4803-8758-945922651515" containerName="console" containerID="cri-o://05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e" gracePeriod=15 Sep 16 07:35:55.266194 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.266167 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:55.269904 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.269881 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-584ff4576-m57c9" Sep 16 07:35:55.430203 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.430181 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:55.553965 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.553884 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-console-config\") pod \"62844c41-3da9-4803-8758-945922651515\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " Sep 16 07:35:55.553965 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.553932 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-serving-cert\") pod \"62844c41-3da9-4803-8758-945922651515\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " Sep 16 07:35:55.553965 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.553960 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-oauth-serving-cert\") pod \"62844c41-3da9-4803-8758-945922651515\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " Sep 16 07:35:55.554239 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.554033 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-oauth-config\") pod \"62844c41-3da9-4803-8758-945922651515\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " Sep 16 07:35:55.554239 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.554092 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4h9r\" (UniqueName: \"kubernetes.io/projected/62844c41-3da9-4803-8758-945922651515-kube-api-access-k4h9r\") pod \"62844c41-3da9-4803-8758-945922651515\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " Sep 16 07:35:55.554239 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.554117 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-service-ca\") pod \"62844c41-3da9-4803-8758-945922651515\" (UID: \"62844c41-3da9-4803-8758-945922651515\") " Sep 16 07:35:55.554399 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.554343 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "62844c41-3da9-4803-8758-945922651515" (UID: "62844c41-3da9-4803-8758-945922651515"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:55.554399 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.554340 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-console-config" (OuterVolumeSpecName: "console-config") pod "62844c41-3da9-4803-8758-945922651515" (UID: "62844c41-3da9-4803-8758-945922651515"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:55.554622 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.554596 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-service-ca" (OuterVolumeSpecName: "service-ca") pod "62844c41-3da9-4803-8758-945922651515" (UID: "62844c41-3da9-4803-8758-945922651515"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:55.556182 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.556154 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "62844c41-3da9-4803-8758-945922651515" (UID: "62844c41-3da9-4803-8758-945922651515"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:35:55.556331 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.556245 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "62844c41-3da9-4803-8758-945922651515" (UID: "62844c41-3da9-4803-8758-945922651515"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:35:55.556331 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.556246 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62844c41-3da9-4803-8758-945922651515-kube-api-access-k4h9r" (OuterVolumeSpecName: "kube-api-access-k4h9r") pod "62844c41-3da9-4803-8758-945922651515" (UID: "62844c41-3da9-4803-8758-945922651515"). InnerVolumeSpecName "kube-api-access-k4h9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:35:55.655125 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.655095 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4h9r\" (UniqueName: \"kubernetes.io/projected/62844c41-3da9-4803-8758-945922651515-kube-api-access-k4h9r\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:55.655125 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.655120 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:55.655355 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.655134 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-console-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:55.655355 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.655146 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-serving-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:55.655355 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.655157 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62844c41-3da9-4803-8758-945922651515-oauth-serving-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:55.655355 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:55.655168 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62844c41-3da9-4803-8758-945922651515-console-oauth-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:35:56.222289 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:56.222247 2580 generic.go:356] "Generic (PLEG): container finished" podID="62844c41-3da9-4803-8758-945922651515" containerID="05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e" exitCode=0 Sep 16 07:35:56.222654 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:56.222306 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856c6f86fd-sbzj2" Sep 16 07:35:56.222654 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:56.222337 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856c6f86fd-sbzj2" event={"ID":"62844c41-3da9-4803-8758-945922651515","Type":"ContainerDied","Data":"05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e"} Sep 16 07:35:56.222654 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:56.222391 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856c6f86fd-sbzj2" event={"ID":"62844c41-3da9-4803-8758-945922651515","Type":"ContainerDied","Data":"b644019767858b2279b15cb3d57fa0ba5e1351a48b9270cb60069fc635e3725c"} Sep 16 07:35:56.222654 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:56.222411 2580 scope.go:117] "RemoveContainer" containerID="05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e" Sep 16 07:35:56.230649 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:56.230632 2580 scope.go:117] "RemoveContainer" containerID="05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e" Sep 16 07:35:56.230919 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:35:56.230901 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e\": container with ID starting with 05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e not found: ID does not exist" containerID="05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e" Sep 16 07:35:56.230976 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:56.230927 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e"} err="failed to get container status \"05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e\": rpc error: code = NotFound desc = could not find container \"05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e\": container with ID starting with 05e074075d4f529982ed5f43bfba01e556991f93249cd486add0a07a564bba6e not found: ID does not exist" Sep 16 07:35:56.236726 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:56.236704 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-856c6f86fd-sbzj2"] Sep 16 07:35:56.238916 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:56.238894 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-856c6f86fd-sbzj2"] Sep 16 07:35:57.228701 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:57.228671 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a8af6d2-1836-4dc6-81c3-85fc2466ba2d" containerID="11d304d2bce61982ed3a148e05d7f9c251a4240e686a34ca3a934015620f47fc" exitCode=0 Sep 16 07:35:57.229094 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:57.228717 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vfmqh" event={"ID":"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d","Type":"ContainerDied","Data":"11d304d2bce61982ed3a148e05d7f9c251a4240e686a34ca3a934015620f47fc"} Sep 16 07:35:57.229094 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:57.229031 2580 scope.go:117] "RemoveContainer" containerID="11d304d2bce61982ed3a148e05d7f9c251a4240e686a34ca3a934015620f47fc" Sep 16 07:35:57.558693 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:57.558607 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62844c41-3da9-4803-8758-945922651515" path="/var/lib/kubelet/pods/62844c41-3da9-4803-8758-945922651515/volumes" Sep 16 07:35:58.234625 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:35:58.234590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vfmqh" event={"ID":"3a8af6d2-1836-4dc6-81c3-85fc2466ba2d","Type":"ContainerStarted","Data":"cb9d3334f9fe344da895cca240cea94bfa0177fe12b67572ff5ca29e2c4a1a78"} Sep 16 07:36:07.270354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:07.270321 2580 generic.go:356] "Generic (PLEG): container finished" podID="6f8ff03a-7366-4924-a639-ec3d9e63c398" containerID="5802b3650512b8c91b4ef7cc302981171fa09b73884c8418fe54f3ef2311565c" exitCode=0 Sep 16 07:36:07.270722 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:07.270394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" event={"ID":"6f8ff03a-7366-4924-a639-ec3d9e63c398","Type":"ContainerDied","Data":"5802b3650512b8c91b4ef7cc302981171fa09b73884c8418fe54f3ef2311565c"} Sep 16 07:36:07.270722 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:07.270698 2580 scope.go:117] "RemoveContainer" containerID="5802b3650512b8c91b4ef7cc302981171fa09b73884c8418fe54f3ef2311565c" Sep 16 07:36:08.277841 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:08.277804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9rgl4" event={"ID":"6f8ff03a-7366-4924-a639-ec3d9e63c398","Type":"ContainerStarted","Data":"c5be6f84ef5988447529107e6d7e0292a884fcc5a32bd33f878e3429c96cb962"} Sep 16 07:36:38.586522 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.586489 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c585d7bb9-4jhck"] Sep 16 07:36:38.586983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.586791 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62844c41-3da9-4803-8758-945922651515" containerName="console" Sep 16 07:36:38.586983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.586803 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62844c41-3da9-4803-8758-945922651515" containerName="console" Sep 16 07:36:38.586983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.586812 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" containerName="registry" Sep 16 07:36:38.586983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.586817 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" containerName="registry" Sep 16 07:36:38.586983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.586830 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22c01a0-e8d3-477a-9655-584f7d204eeb" containerName="console" Sep 16 07:36:38.586983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.586835 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22c01a0-e8d3-477a-9655-584f7d204eeb" containerName="console" Sep 16 07:36:38.586983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.586882 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62844c41-3da9-4803-8758-945922651515" containerName="console" Sep 16 07:36:38.586983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.586889 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbfe1cf4-6dcd-424d-af1c-cb8d82c0b7fa" containerName="registry" Sep 16 07:36:38.586983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.586896 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22c01a0-e8d3-477a-9655-584f7d204eeb" containerName="console" Sep 16 07:36:38.591824 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.591799 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.600881 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.600854 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c585d7bb9-4jhck"] Sep 16 07:36:38.682096 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.682068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-serving-cert\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.682096 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.682096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-console-config\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.682349 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.682118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxlw\" (UniqueName: \"kubernetes.io/projected/949734a3-644d-4956-b513-8d7df9a6b084-kube-api-access-wgxlw\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.682349 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.682145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-service-ca\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.682349 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.682265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-oauth-serving-cert\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.682349 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.682306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-oauth-config\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.682530 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.682401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-trusted-ca-bundle\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.783423 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.783378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-service-ca\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.783576 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.783442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-oauth-serving-cert\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.783576 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.783470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-oauth-config\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.783576 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.783542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-trusted-ca-bundle\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.783734 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.783578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-serving-cert\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.783734 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.783601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-console-config\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.783734 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.783638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxlw\" (UniqueName: \"kubernetes.io/projected/949734a3-644d-4956-b513-8d7df9a6b084-kube-api-access-wgxlw\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.784248 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.784194 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-service-ca\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.784422 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.784194 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-oauth-serving-cert\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.784422 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.784316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-console-config\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.784701 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.784682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-trusted-ca-bundle\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.786013 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.785987 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-serving-cert\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.786086 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.786048 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-oauth-config\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.791440 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.791419 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxlw\" (UniqueName: \"kubernetes.io/projected/949734a3-644d-4956-b513-8d7df9a6b084-kube-api-access-wgxlw\") pod \"console-5c585d7bb9-4jhck\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:38.903618 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:38.903589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:39.174407 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:39.174381 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c585d7bb9-4jhck"] Sep 16 07:36:39.176779 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:36:39.176751 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949734a3_644d_4956_b513_8d7df9a6b084.slice/crio-7803fb2f04b0aeddc260f61f2436f2a2164c8fde1a56f8fd0f6b92a5b5d6da6c WatchSource:0}: Error finding container 7803fb2f04b0aeddc260f61f2436f2a2164c8fde1a56f8fd0f6b92a5b5d6da6c: Status 404 returned error can't find the container with id 7803fb2f04b0aeddc260f61f2436f2a2164c8fde1a56f8fd0f6b92a5b5d6da6c Sep 16 07:36:39.397978 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:39.397941 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c585d7bb9-4jhck" event={"ID":"949734a3-644d-4956-b513-8d7df9a6b084","Type":"ContainerStarted","Data":"263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285"} Sep 16 07:36:39.398131 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:39.397991 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c585d7bb9-4jhck" event={"ID":"949734a3-644d-4956-b513-8d7df9a6b084","Type":"ContainerStarted","Data":"7803fb2f04b0aeddc260f61f2436f2a2164c8fde1a56f8fd0f6b92a5b5d6da6c"} Sep 16 07:36:39.414360 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:39.414314 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c585d7bb9-4jhck" podStartSLOduration=1.414294016 podStartE2EDuration="1.414294016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:36:39.41359371 +0000 UTC m=+172.333209658" watchObservedRunningTime="2026-09-16 07:36:39.414294016 +0000 UTC m=+172.333909963" Sep 16 07:36:48.904646 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:48.904607 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:48.905013 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:48.904670 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:48.909433 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:48.909405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:49.439577 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:49.439547 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:36:49.483155 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:36:49.483122 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c5577f8c9-fvpnq"] Sep 16 07:37:14.505178 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.505131 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c5577f8c9-fvpnq" podUID="30ab6452-4aa4-45ed-b22d-9ab75df94a99" containerName="console" containerID="cri-o://d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c" gracePeriod=15 Sep 16 07:37:14.742191 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.742170 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:37:14.825510 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.825452 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-oauth-config\") pod \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " Sep 16 07:37:14.825510 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.825485 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-config\") pod \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " Sep 16 07:37:14.825679 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.825519 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-service-ca\") pod \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " Sep 16 07:37:14.825679 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.825537 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-serving-cert\") pod \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " Sep 16 07:37:14.825679 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.825559 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92s77\" (UniqueName: \"kubernetes.io/projected/30ab6452-4aa4-45ed-b22d-9ab75df94a99-kube-api-access-92s77\") pod \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " Sep 16 07:37:14.825679 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.825577 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-oauth-serving-cert\") pod \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " Sep 16 07:37:14.825883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.825703 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-trusted-ca-bundle\") pod \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\" (UID: \"30ab6452-4aa4-45ed-b22d-9ab75df94a99\") " Sep 16 07:37:14.825950 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.825922 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-service-ca" (OuterVolumeSpecName: "service-ca") pod "30ab6452-4aa4-45ed-b22d-9ab75df94a99" (UID: "30ab6452-4aa4-45ed-b22d-9ab75df94a99"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:37:14.826008 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.825969 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "30ab6452-4aa4-45ed-b22d-9ab75df94a99" (UID: "30ab6452-4aa4-45ed-b22d-9ab75df94a99"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:37:14.826008 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.825994 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-config" (OuterVolumeSpecName: "console-config") pod "30ab6452-4aa4-45ed-b22d-9ab75df94a99" (UID: "30ab6452-4aa4-45ed-b22d-9ab75df94a99"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:37:14.826094 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.826041 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "30ab6452-4aa4-45ed-b22d-9ab75df94a99" (UID: "30ab6452-4aa4-45ed-b22d-9ab75df94a99"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:37:14.826374 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.826354 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:37:14.826422 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.826382 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-oauth-serving-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:37:14.826422 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.826398 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-trusted-ca-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:37:14.826422 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.826412 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:37:14.827639 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.827606 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "30ab6452-4aa4-45ed-b22d-9ab75df94a99" (UID: "30ab6452-4aa4-45ed-b22d-9ab75df94a99"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:37:14.827734 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.827658 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "30ab6452-4aa4-45ed-b22d-9ab75df94a99" (UID: "30ab6452-4aa4-45ed-b22d-9ab75df94a99"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:37:14.827734 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.827659 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ab6452-4aa4-45ed-b22d-9ab75df94a99-kube-api-access-92s77" (OuterVolumeSpecName: "kube-api-access-92s77") pod "30ab6452-4aa4-45ed-b22d-9ab75df94a99" (UID: "30ab6452-4aa4-45ed-b22d-9ab75df94a99"). InnerVolumeSpecName "kube-api-access-92s77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:37:14.927094 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.927069 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-oauth-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:37:14.927094 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.927090 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30ab6452-4aa4-45ed-b22d-9ab75df94a99-console-serving-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:37:14.927246 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:14.927100 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92s77\" (UniqueName: \"kubernetes.io/projected/30ab6452-4aa4-45ed-b22d-9ab75df94a99-kube-api-access-92s77\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:37:15.539740 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:15.539692 2580 generic.go:356] "Generic (PLEG): container finished" podID="30ab6452-4aa4-45ed-b22d-9ab75df94a99" containerID="d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c" exitCode=0 Sep 16 07:37:15.540125 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:15.539748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5577f8c9-fvpnq" event={"ID":"30ab6452-4aa4-45ed-b22d-9ab75df94a99","Type":"ContainerDied","Data":"d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c"} Sep 16 07:37:15.540125 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:15.539756 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5577f8c9-fvpnq" Sep 16 07:37:15.540125 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:15.539800 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5577f8c9-fvpnq" event={"ID":"30ab6452-4aa4-45ed-b22d-9ab75df94a99","Type":"ContainerDied","Data":"18481b8a81d19bb24d896385d6ce6fda081e1a3be3055133120351c2d3affb24"} Sep 16 07:37:15.540125 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:15.539827 2580 scope.go:117] "RemoveContainer" containerID="d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c" Sep 16 07:37:15.548967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:15.548948 2580 scope.go:117] "RemoveContainer" containerID="d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c" Sep 16 07:37:15.549252 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:37:15.549232 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c\": container with ID starting with d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c not found: ID does not exist" containerID="d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c" Sep 16 07:37:15.549319 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:15.549259 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c"} err="failed to get container status \"d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c\": rpc error: code = NotFound desc = could not find container \"d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c\": container with ID starting with d137502464aee8d2ab89e20c2ccb8567e95c4f5b70ff264a7215abf3cd99fa0c not found: ID does not exist" Sep 16 07:37:15.561641 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:15.561616 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c5577f8c9-fvpnq"] Sep 16 07:37:15.564707 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:15.564683 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c5577f8c9-fvpnq"] Sep 16 07:37:17.558421 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:17.558382 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ab6452-4aa4-45ed-b22d-9ab75df94a99" path="/var/lib/kubelet/pods/30ab6452-4aa4-45ed-b22d-9ab75df94a99/volumes" Sep 16 07:37:49.531166 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.531131 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-df8c4cb68-lbjqp"] Sep 16 07:37:49.531706 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.531635 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ab6452-4aa4-45ed-b22d-9ab75df94a99" containerName="console" Sep 16 07:37:49.531706 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.531658 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ab6452-4aa4-45ed-b22d-9ab75df94a99" containerName="console" Sep 16 07:37:49.531824 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.531759 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ab6452-4aa4-45ed-b22d-9ab75df94a99" containerName="console" Sep 16 07:37:49.536233 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.536190 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.545746 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.545720 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df8c4cb68-lbjqp"] Sep 16 07:37:49.683082 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.683049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-console-config\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.683082 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.683080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-trusted-ca-bundle\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.683329 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.683100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-oauth-config\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.683329 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.683131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-oauth-serving-cert\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.683329 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.683194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-serving-cert\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.683329 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.683265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kmdr\" (UniqueName: \"kubernetes.io/projected/f01e631b-65db-459f-892a-9c32cf1e9e46-kube-api-access-5kmdr\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.683329 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.683285 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-service-ca\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.784380 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.784306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-oauth-serving-cert\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.784380 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.784357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-serving-cert\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.784563 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.784413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kmdr\" (UniqueName: \"kubernetes.io/projected/f01e631b-65db-459f-892a-9c32cf1e9e46-kube-api-access-5kmdr\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.784563 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.784455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-service-ca\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.784563 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.784498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-console-config\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.784563 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.784523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-trusted-ca-bundle\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.784563 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.784558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-oauth-config\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.785151 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.785118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-service-ca\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.785294 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.785121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-oauth-serving-cert\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.785346 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.785300 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-console-config\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.785464 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.785443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-trusted-ca-bundle\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.787401 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.787382 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-serving-cert\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.787508 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.787488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-oauth-config\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.792142 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.792123 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kmdr\" (UniqueName: \"kubernetes.io/projected/f01e631b-65db-459f-892a-9c32cf1e9e46-kube-api-access-5kmdr\") pod \"console-df8c4cb68-lbjqp\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:49.847303 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:49.847279 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:50.115941 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:50.115918 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df8c4cb68-lbjqp"] Sep 16 07:37:50.118119 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:37:50.118085 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01e631b_65db_459f_892a_9c32cf1e9e46.slice/crio-1747fb1cdc929c4af810814f45dd8a9c8458dac54308caae79a226da38128eb2 WatchSource:0}: Error finding container 1747fb1cdc929c4af810814f45dd8a9c8458dac54308caae79a226da38128eb2: Status 404 returned error can't find the container with id 1747fb1cdc929c4af810814f45dd8a9c8458dac54308caae79a226da38128eb2 Sep 16 07:37:50.686392 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:50.686356 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df8c4cb68-lbjqp" event={"ID":"f01e631b-65db-459f-892a-9c32cf1e9e46","Type":"ContainerStarted","Data":"4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f"} Sep 16 07:37:50.686834 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:50.686425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df8c4cb68-lbjqp" event={"ID":"f01e631b-65db-459f-892a-9c32cf1e9e46","Type":"ContainerStarted","Data":"1747fb1cdc929c4af810814f45dd8a9c8458dac54308caae79a226da38128eb2"} Sep 16 07:37:50.703610 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:50.703559 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-df8c4cb68-lbjqp" podStartSLOduration=1.703545184 podStartE2EDuration="1.703545184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:37:50.702296722 +0000 UTC m=+243.621912667" watchObservedRunningTime="2026-09-16 07:37:50.703545184 +0000 UTC m=+243.623161197" Sep 16 07:37:59.848321 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:59.848280 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:59.848690 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:59.848346 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:37:59.853367 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:37:59.853339 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:38:00.732535 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:00.732505 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:38:00.775406 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:00.775370 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c585d7bb9-4jhck"] Sep 16 07:38:25.796587 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:25.796490 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c585d7bb9-4jhck" podUID="949734a3-644d-4956-b513-8d7df9a6b084" containerName="console" containerID="cri-o://263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285" gracePeriod=15 Sep 16 07:38:26.024898 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.024871 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:38:26.066402 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.066331 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgxlw\" (UniqueName: \"kubernetes.io/projected/949734a3-644d-4956-b513-8d7df9a6b084-kube-api-access-wgxlw\") pod \"949734a3-644d-4956-b513-8d7df9a6b084\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " Sep 16 07:38:26.066402 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.066377 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-serving-cert\") pod \"949734a3-644d-4956-b513-8d7df9a6b084\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " Sep 16 07:38:26.066648 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.066420 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-trusted-ca-bundle\") pod \"949734a3-644d-4956-b513-8d7df9a6b084\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " Sep 16 07:38:26.066648 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.066535 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-service-ca\") pod \"949734a3-644d-4956-b513-8d7df9a6b084\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " Sep 16 07:38:26.066648 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.066563 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-console-config\") pod \"949734a3-644d-4956-b513-8d7df9a6b084\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " Sep 16 07:38:26.066648 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.066590 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-oauth-serving-cert\") pod \"949734a3-644d-4956-b513-8d7df9a6b084\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " Sep 16 07:38:26.066648 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.066616 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-oauth-config\") pod \"949734a3-644d-4956-b513-8d7df9a6b084\" (UID: \"949734a3-644d-4956-b513-8d7df9a6b084\") " Sep 16 07:38:26.066947 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.066912 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-service-ca" (OuterVolumeSpecName: "service-ca") pod "949734a3-644d-4956-b513-8d7df9a6b084" (UID: "949734a3-644d-4956-b513-8d7df9a6b084"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:38:26.067009 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.066934 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "949734a3-644d-4956-b513-8d7df9a6b084" (UID: "949734a3-644d-4956-b513-8d7df9a6b084"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:38:26.067230 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.067192 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:38:26.067314 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.067250 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-trusted-ca-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:38:26.067639 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.067602 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-console-config" (OuterVolumeSpecName: "console-config") pod "949734a3-644d-4956-b513-8d7df9a6b084" (UID: "949734a3-644d-4956-b513-8d7df9a6b084"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:38:26.067937 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.067905 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "949734a3-644d-4956-b513-8d7df9a6b084" (UID: "949734a3-644d-4956-b513-8d7df9a6b084"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:38:26.069463 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.069427 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "949734a3-644d-4956-b513-8d7df9a6b084" (UID: "949734a3-644d-4956-b513-8d7df9a6b084"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:38:26.069665 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.069636 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949734a3-644d-4956-b513-8d7df9a6b084-kube-api-access-wgxlw" (OuterVolumeSpecName: "kube-api-access-wgxlw") pod "949734a3-644d-4956-b513-8d7df9a6b084" (UID: "949734a3-644d-4956-b513-8d7df9a6b084"). InnerVolumeSpecName "kube-api-access-wgxlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:38:26.072898 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.072877 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "949734a3-644d-4956-b513-8d7df9a6b084" (UID: "949734a3-644d-4956-b513-8d7df9a6b084"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:38:26.167778 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.167759 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-console-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:38:26.167778 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.167778 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/949734a3-644d-4956-b513-8d7df9a6b084-oauth-serving-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:38:26.167896 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.167789 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-oauth-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:38:26.167896 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.167799 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgxlw\" (UniqueName: \"kubernetes.io/projected/949734a3-644d-4956-b513-8d7df9a6b084-kube-api-access-wgxlw\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:38:26.167896 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.167807 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/949734a3-644d-4956-b513-8d7df9a6b084-console-serving-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:38:26.833482 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.833450 2580 generic.go:356] "Generic (PLEG): container finished" podID="949734a3-644d-4956-b513-8d7df9a6b084" containerID="263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285" exitCode=0 Sep 16 07:38:26.833885 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.833494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c585d7bb9-4jhck" event={"ID":"949734a3-644d-4956-b513-8d7df9a6b084","Type":"ContainerDied","Data":"263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285"} Sep 16 07:38:26.833885 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.833530 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c585d7bb9-4jhck" event={"ID":"949734a3-644d-4956-b513-8d7df9a6b084","Type":"ContainerDied","Data":"7803fb2f04b0aeddc260f61f2436f2a2164c8fde1a56f8fd0f6b92a5b5d6da6c"} Sep 16 07:38:26.833885 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.833550 2580 scope.go:117] "RemoveContainer" containerID="263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285" Sep 16 07:38:26.833885 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.833496 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c585d7bb9-4jhck" Sep 16 07:38:26.842605 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.842588 2580 scope.go:117] "RemoveContainer" containerID="263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285" Sep 16 07:38:26.842900 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:38:26.842879 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285\": container with ID starting with 263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285 not found: ID does not exist" containerID="263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285" Sep 16 07:38:26.842973 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.842907 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285"} err="failed to get container status \"263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285\": rpc error: code = NotFound desc = could not find container \"263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285\": container with ID starting with 263045c137e6141b17bf5ce2b88337625bffcd4999f607eabc59dee06a818285 not found: ID does not exist" Sep 16 07:38:26.855724 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.855698 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c585d7bb9-4jhck"] Sep 16 07:38:26.860462 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:26.860437 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c585d7bb9-4jhck"] Sep 16 07:38:27.558219 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:27.558187 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="949734a3-644d-4956-b513-8d7df9a6b084" path="/var/lib/kubelet/pods/949734a3-644d-4956-b513-8d7df9a6b084/volumes" Sep 16 07:38:47.420604 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:47.420578 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:38:47.421926 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:47.421905 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:38:47.432410 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:38:47.432390 2580 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 07:39:04.332991 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.332959 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx"] Sep 16 07:39:04.335261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.333292 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="949734a3-644d-4956-b513-8d7df9a6b084" containerName="console" Sep 16 07:39:04.335261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.333305 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="949734a3-644d-4956-b513-8d7df9a6b084" containerName="console" Sep 16 07:39:04.335261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.333355 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="949734a3-644d-4956-b513-8d7df9a6b084" containerName="console" Sep 16 07:39:04.336198 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.336183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.339095 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.339073 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l72bd\"" Sep 16 07:39:04.339391 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.339377 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 07:39:04.340434 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.340413 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 07:39:04.344400 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.344375 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx"] Sep 16 07:39:04.454486 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.454460 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.454640 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.454500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.454640 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.454585 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxrdb\" (UniqueName: \"kubernetes.io/projected/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-kube-api-access-kxrdb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.555835 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.555808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxrdb\" (UniqueName: \"kubernetes.io/projected/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-kube-api-access-kxrdb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.555911 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.555861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.555911 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.555882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.556299 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.556279 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.556346 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.556297 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.564193 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.564169 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxrdb\" (UniqueName: \"kubernetes.io/projected/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-kube-api-access-kxrdb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.645740 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.645689 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:04.916447 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.916422 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx"] Sep 16 07:39:04.918906 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:39:04.918879 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fd125b_0d7f_4e81_b052_4d5e429a6dc3.slice/crio-3c4dc7141ae30fec65795ae8e227871d2895eea0dc508e45cab71e327f13285e WatchSource:0}: Error finding container 3c4dc7141ae30fec65795ae8e227871d2895eea0dc508e45cab71e327f13285e: Status 404 returned error can't find the container with id 3c4dc7141ae30fec65795ae8e227871d2895eea0dc508e45cab71e327f13285e Sep 16 07:39:04.920968 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.920950 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:39:04.984387 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:04.984359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" event={"ID":"22fd125b-0d7f-4e81-b052-4d5e429a6dc3","Type":"ContainerStarted","Data":"3c4dc7141ae30fec65795ae8e227871d2895eea0dc508e45cab71e327f13285e"} Sep 16 07:39:12.022622 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:12.022594 2580 generic.go:356] "Generic (PLEG): container finished" podID="22fd125b-0d7f-4e81-b052-4d5e429a6dc3" containerID="6503913ad9600c006b39b38808a2aa3ccef77ba705498c1cad14594daab3f897" exitCode=0 Sep 16 07:39:12.022979 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:12.022676 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" event={"ID":"22fd125b-0d7f-4e81-b052-4d5e429a6dc3","Type":"ContainerDied","Data":"6503913ad9600c006b39b38808a2aa3ccef77ba705498c1cad14594daab3f897"} Sep 16 07:39:15.038116 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:15.038079 2580 generic.go:356] "Generic (PLEG): container finished" podID="22fd125b-0d7f-4e81-b052-4d5e429a6dc3" containerID="bff88192e1c6ad2d31217468c8d7c740daa95bf40aed7ce26faa46a5867c2d65" exitCode=0 Sep 16 07:39:15.038602 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:15.038171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" event={"ID":"22fd125b-0d7f-4e81-b052-4d5e429a6dc3","Type":"ContainerDied","Data":"bff88192e1c6ad2d31217468c8d7c740daa95bf40aed7ce26faa46a5867c2d65"} Sep 16 07:39:22.073116 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:22.073080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" event={"ID":"22fd125b-0d7f-4e81-b052-4d5e429a6dc3","Type":"ContainerStarted","Data":"ca0544d85886bd367b53c950602efb03439820cbe3dc8bbc31dbacd3f5df022c"} Sep 16 07:39:22.091275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:22.091229 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" podStartSLOduration=2.093800754 podStartE2EDuration="18.091194691s" totalImagesPullingTime="15.997393937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:39:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:39:22.088590417 +0000 UTC m=+335.008206365" watchObservedRunningTime="2026-09-16 07:39:22.091194691 +0000 UTC m=+335.010810637" Sep 16 07:39:23.079140 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:23.079106 2580 generic.go:356] "Generic (PLEG): container finished" podID="22fd125b-0d7f-4e81-b052-4d5e429a6dc3" containerID="ca0544d85886bd367b53c950602efb03439820cbe3dc8bbc31dbacd3f5df022c" exitCode=0 Sep 16 07:39:23.079535 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:23.079180 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" event={"ID":"22fd125b-0d7f-4e81-b052-4d5e429a6dc3","Type":"ContainerDied","Data":"ca0544d85886bd367b53c950602efb03439820cbe3dc8bbc31dbacd3f5df022c"} Sep 16 07:39:24.203743 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:24.203722 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:24.217686 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:24.217663 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-util\") pod \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " Sep 16 07:39:24.217785 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:24.217705 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxrdb\" (UniqueName: \"kubernetes.io/projected/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-kube-api-access-kxrdb\") pod \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " Sep 16 07:39:24.217785 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:24.217758 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-bundle\") pod \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\" (UID: \"22fd125b-0d7f-4e81-b052-4d5e429a6dc3\") " Sep 16 07:39:24.218462 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:24.218434 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-bundle" (OuterVolumeSpecName: "bundle") pod "22fd125b-0d7f-4e81-b052-4d5e429a6dc3" (UID: "22fd125b-0d7f-4e81-b052-4d5e429a6dc3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:39:24.219812 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:24.219785 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-kube-api-access-kxrdb" (OuterVolumeSpecName: "kube-api-access-kxrdb") pod "22fd125b-0d7f-4e81-b052-4d5e429a6dc3" (UID: "22fd125b-0d7f-4e81-b052-4d5e429a6dc3"). InnerVolumeSpecName "kube-api-access-kxrdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:39:24.222713 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:24.222686 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-util" (OuterVolumeSpecName: "util") pod "22fd125b-0d7f-4e81-b052-4d5e429a6dc3" (UID: "22fd125b-0d7f-4e81-b052-4d5e429a6dc3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:39:24.319023 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:24.318998 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxrdb\" (UniqueName: \"kubernetes.io/projected/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-kube-api-access-kxrdb\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:39:24.319023 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:24.319019 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:39:24.319176 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:24.319028 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22fd125b-0d7f-4e81-b052-4d5e429a6dc3-util\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:39:25.089837 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:25.089805 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" Sep 16 07:39:25.090018 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:25.089807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s2mqx" event={"ID":"22fd125b-0d7f-4e81-b052-4d5e429a6dc3","Type":"ContainerDied","Data":"3c4dc7141ae30fec65795ae8e227871d2895eea0dc508e45cab71e327f13285e"} Sep 16 07:39:25.090018 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:25.089935 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c4dc7141ae30fec65795ae8e227871d2895eea0dc508e45cab71e327f13285e" Sep 16 07:39:32.495982 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.495948 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95"] Sep 16 07:39:32.496461 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.496254 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fd125b-0d7f-4e81-b052-4d5e429a6dc3" containerName="pull" Sep 16 07:39:32.496461 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.496265 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fd125b-0d7f-4e81-b052-4d5e429a6dc3" containerName="pull" Sep 16 07:39:32.496461 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.496274 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fd125b-0d7f-4e81-b052-4d5e429a6dc3" containerName="extract" Sep 16 07:39:32.496461 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.496281 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fd125b-0d7f-4e81-b052-4d5e429a6dc3" containerName="extract" Sep 16 07:39:32.496461 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.496292 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fd125b-0d7f-4e81-b052-4d5e429a6dc3" containerName="util" Sep 16 07:39:32.496461 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.496298 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fd125b-0d7f-4e81-b052-4d5e429a6dc3" containerName="util" Sep 16 07:39:32.496461 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.496367 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="22fd125b-0d7f-4e81-b052-4d5e429a6dc3" containerName="extract" Sep 16 07:39:32.546102 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.546073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95"] Sep 16 07:39:32.546257 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.546163 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" Sep 16 07:39:32.548897 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.548875 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-q9c2d\"" Sep 16 07:39:32.549044 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.548907 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 16 07:39:32.549109 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.549072 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:39:32.583055 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.583036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/808eacc4-0b9a-45cf-bb64-892735a4b57e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wbq95\" (UID: \"808eacc4-0b9a-45cf-bb64-892735a4b57e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" Sep 16 07:39:32.583168 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.583083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pckh\" (UniqueName: \"kubernetes.io/projected/808eacc4-0b9a-45cf-bb64-892735a4b57e-kube-api-access-7pckh\") pod \"cert-manager-operator-controller-manager-55c444bf86-wbq95\" (UID: \"808eacc4-0b9a-45cf-bb64-892735a4b57e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" Sep 16 07:39:32.683860 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.683831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/808eacc4-0b9a-45cf-bb64-892735a4b57e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wbq95\" (UID: \"808eacc4-0b9a-45cf-bb64-892735a4b57e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" Sep 16 07:39:32.683958 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.683874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pckh\" (UniqueName: \"kubernetes.io/projected/808eacc4-0b9a-45cf-bb64-892735a4b57e-kube-api-access-7pckh\") pod \"cert-manager-operator-controller-manager-55c444bf86-wbq95\" (UID: \"808eacc4-0b9a-45cf-bb64-892735a4b57e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" Sep 16 07:39:32.684268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.684252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/808eacc4-0b9a-45cf-bb64-892735a4b57e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wbq95\" (UID: \"808eacc4-0b9a-45cf-bb64-892735a4b57e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" Sep 16 07:39:32.694170 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.694150 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pckh\" (UniqueName: \"kubernetes.io/projected/808eacc4-0b9a-45cf-bb64-892735a4b57e-kube-api-access-7pckh\") pod \"cert-manager-operator-controller-manager-55c444bf86-wbq95\" (UID: \"808eacc4-0b9a-45cf-bb64-892735a4b57e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" Sep 16 07:39:32.855679 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:32.855625 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" Sep 16 07:39:33.150883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:33.150807 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95"] Sep 16 07:39:33.154046 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:39:33.154018 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808eacc4_0b9a_45cf_bb64_892735a4b57e.slice/crio-ebd6ddd6057ecdef4c43fd25844f87843f41c3dbca673dd0ad03be8084850669 WatchSource:0}: Error finding container ebd6ddd6057ecdef4c43fd25844f87843f41c3dbca673dd0ad03be8084850669: Status 404 returned error can't find the container with id ebd6ddd6057ecdef4c43fd25844f87843f41c3dbca673dd0ad03be8084850669 Sep 16 07:39:34.126861 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:34.126827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" event={"ID":"808eacc4-0b9a-45cf-bb64-892735a4b57e","Type":"ContainerStarted","Data":"ebd6ddd6057ecdef4c43fd25844f87843f41c3dbca673dd0ad03be8084850669"} Sep 16 07:39:36.137753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.137721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" event={"ID":"808eacc4-0b9a-45cf-bb64-892735a4b57e","Type":"ContainerStarted","Data":"f78196c59b9fa2c03da1fd3d2b592b6b6c5acd54a35bd07c8c510e6e7d3d6030"} Sep 16 07:39:36.159647 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.159594 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wbq95" podStartSLOduration=1.944030006 podStartE2EDuration="4.159574806s" totalImagesPullingTime="2.2155448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:39:36.158264067 +0000 UTC m=+349.077880013" watchObservedRunningTime="2026-09-16 07:39:36.159574806 +0000 UTC m=+349.079190752" Sep 16 07:39:36.212253 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.212204 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fdkm4"] Sep 16 07:39:36.214421 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.214401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" Sep 16 07:39:36.217181 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.217166 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-wtbqp\"" Sep 16 07:39:36.217257 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.217186 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 07:39:36.217320 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.217301 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 07:39:36.223875 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.223854 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fdkm4"] Sep 16 07:39:36.312401 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.312367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4jm7\" (UniqueName: \"kubernetes.io/projected/5d9dc730-a3fb-4b93-b94c-638cf1c2f60b-kube-api-access-v4jm7\") pod \"cert-manager-webhook-6486ffd997-fdkm4\" (UID: \"5d9dc730-a3fb-4b93-b94c-638cf1c2f60b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" Sep 16 07:39:36.312539 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.312423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d9dc730-a3fb-4b93-b94c-638cf1c2f60b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fdkm4\" (UID: \"5d9dc730-a3fb-4b93-b94c-638cf1c2f60b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" Sep 16 07:39:36.413737 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.413704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4jm7\" (UniqueName: \"kubernetes.io/projected/5d9dc730-a3fb-4b93-b94c-638cf1c2f60b-kube-api-access-v4jm7\") pod \"cert-manager-webhook-6486ffd997-fdkm4\" (UID: \"5d9dc730-a3fb-4b93-b94c-638cf1c2f60b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" Sep 16 07:39:36.413858 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.413774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d9dc730-a3fb-4b93-b94c-638cf1c2f60b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fdkm4\" (UID: \"5d9dc730-a3fb-4b93-b94c-638cf1c2f60b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" Sep 16 07:39:36.431022 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.431001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4jm7\" (UniqueName: \"kubernetes.io/projected/5d9dc730-a3fb-4b93-b94c-638cf1c2f60b-kube-api-access-v4jm7\") pod \"cert-manager-webhook-6486ffd997-fdkm4\" (UID: \"5d9dc730-a3fb-4b93-b94c-638cf1c2f60b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" Sep 16 07:39:36.431022 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.431010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d9dc730-a3fb-4b93-b94c-638cf1c2f60b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fdkm4\" (UID: \"5d9dc730-a3fb-4b93-b94c-638cf1c2f60b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" Sep 16 07:39:36.539113 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.539088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" Sep 16 07:39:36.829689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.829661 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fdkm4"] Sep 16 07:39:36.831414 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:39:36.831387 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d9dc730_a3fb_4b93_b94c_638cf1c2f60b.slice/crio-f6abcbadb842613b8f0a294187dfb880a4d46f3c7d103de6c653d31024013ff1 WatchSource:0}: Error finding container f6abcbadb842613b8f0a294187dfb880a4d46f3c7d103de6c653d31024013ff1: Status 404 returned error can't find the container with id f6abcbadb842613b8f0a294187dfb880a4d46f3c7d103de6c653d31024013ff1 Sep 16 07:39:36.978743 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.978713 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85"] Sep 16 07:39:36.981802 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.981785 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:36.984257 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.984230 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 07:39:36.984398 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.984381 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 07:39:36.984554 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.984535 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l72bd\"" Sep 16 07:39:36.990998 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:36.990966 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85"] Sep 16 07:39:37.019148 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.019125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jr2l\" (UniqueName: \"kubernetes.io/projected/858c823c-5447-4b39-8e6d-4c4f390045c6-kube-api-access-5jr2l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:37.019275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.019174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:37.019275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.019238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:37.120509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.120447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jr2l\" (UniqueName: \"kubernetes.io/projected/858c823c-5447-4b39-8e6d-4c4f390045c6-kube-api-access-5jr2l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:37.120509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.120500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:37.120636 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.120527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:37.120813 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.120795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:37.120850 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.120821 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:37.128150 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.128128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jr2l\" (UniqueName: \"kubernetes.io/projected/858c823c-5447-4b39-8e6d-4c4f390045c6-kube-api-access-5jr2l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:37.143623 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.143599 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" event={"ID":"5d9dc730-a3fb-4b93-b94c-638cf1c2f60b","Type":"ContainerStarted","Data":"f6abcbadb842613b8f0a294187dfb880a4d46f3c7d103de6c653d31024013ff1"} Sep 16 07:39:37.292277 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.292252 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:37.567920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:37.567888 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85"] Sep 16 07:39:37.568033 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:39:37.567926 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858c823c_5447_4b39_8e6d_4c4f390045c6.slice/crio-7720141be6e13add0440c7915694cdc9dff903688fd4caa0b1513380631a8479 WatchSource:0}: Error finding container 7720141be6e13add0440c7915694cdc9dff903688fd4caa0b1513380631a8479: Status 404 returned error can't find the container with id 7720141be6e13add0440c7915694cdc9dff903688fd4caa0b1513380631a8479 Sep 16 07:39:38.150982 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:38.150940 2580 generic.go:356] "Generic (PLEG): container finished" podID="858c823c-5447-4b39-8e6d-4c4f390045c6" containerID="c45469bc4ecc6c42b28f3f491d29747f1b934b63a4d4583ebe664ce3598d0259" exitCode=0 Sep 16 07:39:38.151428 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:38.150990 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" event={"ID":"858c823c-5447-4b39-8e6d-4c4f390045c6","Type":"ContainerDied","Data":"c45469bc4ecc6c42b28f3f491d29747f1b934b63a4d4583ebe664ce3598d0259"} Sep 16 07:39:38.151428 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:38.151050 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" event={"ID":"858c823c-5447-4b39-8e6d-4c4f390045c6","Type":"ContainerStarted","Data":"7720141be6e13add0440c7915694cdc9dff903688fd4caa0b1513380631a8479"} Sep 16 07:39:40.163465 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:40.163374 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" event={"ID":"5d9dc730-a3fb-4b93-b94c-638cf1c2f60b","Type":"ContainerStarted","Data":"59cbf6d7e5ca2045d4d05bbb73059f5deb8833b6d70fb0361bfbea58f8ec8928"} Sep 16 07:39:40.163873 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:40.163526 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" Sep 16 07:39:40.183299 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:40.183250 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" podStartSLOduration=1.567733886 podStartE2EDuration="4.183230487s" totalImagesPullingTime="2.615496601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:39:40.182535497 +0000 UTC m=+353.102151444" watchObservedRunningTime="2026-09-16 07:39:40.183230487 +0000 UTC m=+353.102846425" Sep 16 07:39:41.172526 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:41.172496 2580 generic.go:356] "Generic (PLEG): container finished" podID="858c823c-5447-4b39-8e6d-4c4f390045c6" containerID="86705de5035598583c22fd80979c2c99468f0d52262195514ec9283c089cc0ac" exitCode=0 Sep 16 07:39:41.172967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:41.172589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" event={"ID":"858c823c-5447-4b39-8e6d-4c4f390045c6","Type":"ContainerDied","Data":"86705de5035598583c22fd80979c2c99468f0d52262195514ec9283c089cc0ac"} Sep 16 07:39:42.179614 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:42.179581 2580 generic.go:356] "Generic (PLEG): container finished" podID="858c823c-5447-4b39-8e6d-4c4f390045c6" containerID="0c5567bdd8ae1a5bef36acbb7d7379b6311a7a42e47e9477012f55e40a7689bd" exitCode=0 Sep 16 07:39:42.179972 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:42.179633 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" event={"ID":"858c823c-5447-4b39-8e6d-4c4f390045c6","Type":"ContainerDied","Data":"0c5567bdd8ae1a5bef36acbb7d7379b6311a7a42e47e9477012f55e40a7689bd"} Sep 16 07:39:43.304608 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:43.304585 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:43.370247 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:43.370201 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jr2l\" (UniqueName: \"kubernetes.io/projected/858c823c-5447-4b39-8e6d-4c4f390045c6-kube-api-access-5jr2l\") pod \"858c823c-5447-4b39-8e6d-4c4f390045c6\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " Sep 16 07:39:43.370363 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:43.370318 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-util\") pod \"858c823c-5447-4b39-8e6d-4c4f390045c6\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " Sep 16 07:39:43.370363 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:43.370358 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-bundle\") pod \"858c823c-5447-4b39-8e6d-4c4f390045c6\" (UID: \"858c823c-5447-4b39-8e6d-4c4f390045c6\") " Sep 16 07:39:43.370745 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:43.370720 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-bundle" (OuterVolumeSpecName: "bundle") pod "858c823c-5447-4b39-8e6d-4c4f390045c6" (UID: "858c823c-5447-4b39-8e6d-4c4f390045c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:39:43.372267 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:43.372243 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858c823c-5447-4b39-8e6d-4c4f390045c6-kube-api-access-5jr2l" (OuterVolumeSpecName: "kube-api-access-5jr2l") pod "858c823c-5447-4b39-8e6d-4c4f390045c6" (UID: "858c823c-5447-4b39-8e6d-4c4f390045c6"). InnerVolumeSpecName "kube-api-access-5jr2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:39:43.374862 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:43.374841 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-util" (OuterVolumeSpecName: "util") pod "858c823c-5447-4b39-8e6d-4c4f390045c6" (UID: "858c823c-5447-4b39-8e6d-4c4f390045c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:39:43.470866 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:43.470809 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jr2l\" (UniqueName: \"kubernetes.io/projected/858c823c-5447-4b39-8e6d-4c4f390045c6-kube-api-access-5jr2l\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:39:43.470866 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:43.470831 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-util\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:39:43.470866 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:43.470841 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/858c823c-5447-4b39-8e6d-4c4f390045c6-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:39:44.191181 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:44.191145 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" event={"ID":"858c823c-5447-4b39-8e6d-4c4f390045c6","Type":"ContainerDied","Data":"7720141be6e13add0440c7915694cdc9dff903688fd4caa0b1513380631a8479"} Sep 16 07:39:44.191379 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:44.191191 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7720141be6e13add0440c7915694cdc9dff903688fd4caa0b1513380631a8479" Sep 16 07:39:44.191379 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:44.191196 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fstv85" Sep 16 07:39:46.174339 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:46.174308 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fdkm4" Sep 16 07:39:47.487532 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.487500 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ndxtl"] Sep 16 07:39:47.487883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.487832 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858c823c-5447-4b39-8e6d-4c4f390045c6" containerName="util" Sep 16 07:39:47.487883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.487843 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="858c823c-5447-4b39-8e6d-4c4f390045c6" containerName="util" Sep 16 07:39:47.487883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.487850 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858c823c-5447-4b39-8e6d-4c4f390045c6" containerName="pull" Sep 16 07:39:47.487883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.487856 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="858c823c-5447-4b39-8e6d-4c4f390045c6" containerName="pull" Sep 16 07:39:47.487883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.487873 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858c823c-5447-4b39-8e6d-4c4f390045c6" containerName="extract" Sep 16 07:39:47.487883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.487878 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="858c823c-5447-4b39-8e6d-4c4f390045c6" containerName="extract" Sep 16 07:39:47.488060 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.487929 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="858c823c-5447-4b39-8e6d-4c4f390045c6" containerName="extract" Sep 16 07:39:47.489980 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.489960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ndxtl" Sep 16 07:39:47.492417 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.492397 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qz9vv\"" Sep 16 07:39:47.499625 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.499602 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ndxtl"] Sep 16 07:39:47.602140 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.602114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sr6s\" (UniqueName: \"kubernetes.io/projected/f3492ab3-537d-4c76-919b-25a3ab942a8b-kube-api-access-7sr6s\") pod \"cert-manager-55b4d954d5-ndxtl\" (UID: \"f3492ab3-537d-4c76-919b-25a3ab942a8b\") " pod="cert-manager/cert-manager-55b4d954d5-ndxtl" Sep 16 07:39:47.602282 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.602164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3492ab3-537d-4c76-919b-25a3ab942a8b-bound-sa-token\") pod \"cert-manager-55b4d954d5-ndxtl\" (UID: \"f3492ab3-537d-4c76-919b-25a3ab942a8b\") " pod="cert-manager/cert-manager-55b4d954d5-ndxtl" Sep 16 07:39:47.703073 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.703046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3492ab3-537d-4c76-919b-25a3ab942a8b-bound-sa-token\") pod \"cert-manager-55b4d954d5-ndxtl\" (UID: \"f3492ab3-537d-4c76-919b-25a3ab942a8b\") " pod="cert-manager/cert-manager-55b4d954d5-ndxtl" Sep 16 07:39:47.703190 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.703094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr6s\" (UniqueName: \"kubernetes.io/projected/f3492ab3-537d-4c76-919b-25a3ab942a8b-kube-api-access-7sr6s\") pod \"cert-manager-55b4d954d5-ndxtl\" (UID: \"f3492ab3-537d-4c76-919b-25a3ab942a8b\") " pod="cert-manager/cert-manager-55b4d954d5-ndxtl" Sep 16 07:39:47.711276 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.711242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3492ab3-537d-4c76-919b-25a3ab942a8b-bound-sa-token\") pod \"cert-manager-55b4d954d5-ndxtl\" (UID: \"f3492ab3-537d-4c76-919b-25a3ab942a8b\") " pod="cert-manager/cert-manager-55b4d954d5-ndxtl" Sep 16 07:39:47.711405 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.711373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sr6s\" (UniqueName: \"kubernetes.io/projected/f3492ab3-537d-4c76-919b-25a3ab942a8b-kube-api-access-7sr6s\") pod \"cert-manager-55b4d954d5-ndxtl\" (UID: \"f3492ab3-537d-4c76-919b-25a3ab942a8b\") " pod="cert-manager/cert-manager-55b4d954d5-ndxtl" Sep 16 07:39:47.800828 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:47.800769 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ndxtl" Sep 16 07:39:48.067560 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:48.067535 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ndxtl"] Sep 16 07:39:48.069913 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:39:48.069883 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3492ab3_537d_4c76_919b_25a3ab942a8b.slice/crio-ff77ab2a072facb4406556ab76622a4c41cd3b3bd6e39242c64e92bd8805dab9 WatchSource:0}: Error finding container ff77ab2a072facb4406556ab76622a4c41cd3b3bd6e39242c64e92bd8805dab9: Status 404 returned error can't find the container with id ff77ab2a072facb4406556ab76622a4c41cd3b3bd6e39242c64e92bd8805dab9 Sep 16 07:39:48.212890 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:48.212849 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ndxtl" event={"ID":"f3492ab3-537d-4c76-919b-25a3ab942a8b","Type":"ContainerStarted","Data":"87685a69810bd815d8a3ed4682285b52356faab90056b20f149a96c96f2674c5"} Sep 16 07:39:48.213063 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:48.212899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ndxtl" event={"ID":"f3492ab3-537d-4c76-919b-25a3ab942a8b","Type":"ContainerStarted","Data":"ff77ab2a072facb4406556ab76622a4c41cd3b3bd6e39242c64e92bd8805dab9"} Sep 16 07:39:48.229929 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:48.229882 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ndxtl" podStartSLOduration=1.2298690620000001 podStartE2EDuration="1.229869062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:39:48.227895565 +0000 UTC m=+361.147511511" watchObservedRunningTime="2026-09-16 07:39:48.229869062 +0000 UTC m=+361.149485009" Sep 16 07:39:50.480519 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.480483 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg"] Sep 16 07:39:50.482737 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.482721 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" Sep 16 07:39:50.485412 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.485388 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:39:50.486689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.486667 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 07:39:50.486807 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.486667 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-j2jr9\"" Sep 16 07:39:50.491493 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.491468 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg"] Sep 16 07:39:50.523307 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.523282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndwx\" (UniqueName: \"kubernetes.io/projected/00c65999-a0af-4804-8c78-99171cd155f8-kube-api-access-vndwx\") pod \"openshift-lws-operator-fd8ccff4c-xsxqg\" (UID: \"00c65999-a0af-4804-8c78-99171cd155f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" Sep 16 07:39:50.523405 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.523342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c65999-a0af-4804-8c78-99171cd155f8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xsxqg\" (UID: \"00c65999-a0af-4804-8c78-99171cd155f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" Sep 16 07:39:50.624309 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.624287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c65999-a0af-4804-8c78-99171cd155f8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xsxqg\" (UID: \"00c65999-a0af-4804-8c78-99171cd155f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" Sep 16 07:39:50.624411 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.624330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndwx\" (UniqueName: \"kubernetes.io/projected/00c65999-a0af-4804-8c78-99171cd155f8-kube-api-access-vndwx\") pod \"openshift-lws-operator-fd8ccff4c-xsxqg\" (UID: \"00c65999-a0af-4804-8c78-99171cd155f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" Sep 16 07:39:50.624657 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.624640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c65999-a0af-4804-8c78-99171cd155f8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xsxqg\" (UID: \"00c65999-a0af-4804-8c78-99171cd155f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" Sep 16 07:39:50.632100 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.632076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndwx\" (UniqueName: \"kubernetes.io/projected/00c65999-a0af-4804-8c78-99171cd155f8-kube-api-access-vndwx\") pod \"openshift-lws-operator-fd8ccff4c-xsxqg\" (UID: \"00c65999-a0af-4804-8c78-99171cd155f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" Sep 16 07:39:50.793163 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:50.793109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" Sep 16 07:39:51.062584 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:51.062560 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg"] Sep 16 07:39:51.064221 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:39:51.064186 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00c65999_a0af_4804_8c78_99171cd155f8.slice/crio-9b946a052a4c15aebe348371858e593947ae649ff72b5a98ebae6f65191ac715 WatchSource:0}: Error finding container 9b946a052a4c15aebe348371858e593947ae649ff72b5a98ebae6f65191ac715: Status 404 returned error can't find the container with id 9b946a052a4c15aebe348371858e593947ae649ff72b5a98ebae6f65191ac715 Sep 16 07:39:51.227738 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:51.227704 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" event={"ID":"00c65999-a0af-4804-8c78-99171cd155f8","Type":"ContainerStarted","Data":"9b946a052a4c15aebe348371858e593947ae649ff72b5a98ebae6f65191ac715"} Sep 16 07:39:53.239649 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:53.239614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" event={"ID":"00c65999-a0af-4804-8c78-99171cd155f8","Type":"ContainerStarted","Data":"d6b1a6f805ca9b6214f78a3066097b7531072f0e35bfda3b860431c8646beae4"} Sep 16 07:39:53.255466 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:53.255415 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xsxqg" podStartSLOduration=1.706910824 podStartE2EDuration="3.255395428s" totalImagesPullingTime="1.548484604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:39:53.254738234 +0000 UTC m=+366.174354180" watchObservedRunningTime="2026-09-16 07:39:53.255395428 +0000 UTC m=+366.175011375" Sep 16 07:39:55.796803 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.796768 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l"] Sep 16 07:39:55.799231 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.799196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:55.801762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.801740 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 07:39:55.801874 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.801812 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l72bd\"" Sep 16 07:39:55.802849 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.802828 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 07:39:55.807496 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.807473 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l"] Sep 16 07:39:55.865113 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.865092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:55.865256 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.865147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7jl\" (UniqueName: \"kubernetes.io/projected/6b20c40d-aeae-45ea-8545-2fabe3a2567e-kube-api-access-7l7jl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:55.865256 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.865180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:55.966156 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.966125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7jl\" (UniqueName: \"kubernetes.io/projected/6b20c40d-aeae-45ea-8545-2fabe3a2567e-kube-api-access-7l7jl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:55.966290 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.966172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:55.966290 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.966257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:55.966590 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.966569 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:55.966662 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.966599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:55.973948 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:55.973922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7jl\" (UniqueName: \"kubernetes.io/projected/6b20c40d-aeae-45ea-8545-2fabe3a2567e-kube-api-access-7l7jl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:56.109237 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:56.109169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:39:56.377111 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:56.377090 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l"] Sep 16 07:39:56.378729 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:39:56.378699 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b20c40d_aeae_45ea_8545_2fabe3a2567e.slice/crio-1c1538c33f45cb3428e586af8e57c42e967abc209866be2506947ca79bcfa811 WatchSource:0}: Error finding container 1c1538c33f45cb3428e586af8e57c42e967abc209866be2506947ca79bcfa811: Status 404 returned error can't find the container with id 1c1538c33f45cb3428e586af8e57c42e967abc209866be2506947ca79bcfa811 Sep 16 07:39:57.259149 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:57.259066 2580 generic.go:356] "Generic (PLEG): container finished" podID="6b20c40d-aeae-45ea-8545-2fabe3a2567e" containerID="b8f82ed7f407d4a94866d6fb967a1db473cfd4de7aca0b3399460bcd055f1fae" exitCode=0 Sep 16 07:39:57.259504 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:57.259149 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" event={"ID":"6b20c40d-aeae-45ea-8545-2fabe3a2567e","Type":"ContainerDied","Data":"b8f82ed7f407d4a94866d6fb967a1db473cfd4de7aca0b3399460bcd055f1fae"} Sep 16 07:39:57.259504 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:57.259201 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" event={"ID":"6b20c40d-aeae-45ea-8545-2fabe3a2567e","Type":"ContainerStarted","Data":"1c1538c33f45cb3428e586af8e57c42e967abc209866be2506947ca79bcfa811"} Sep 16 07:39:58.266297 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:58.266266 2580 generic.go:356] "Generic (PLEG): container finished" podID="6b20c40d-aeae-45ea-8545-2fabe3a2567e" containerID="f58f3fcf50046f7cbf251761090c9f62c471aff629ad54706b337f37d2f27e59" exitCode=0 Sep 16 07:39:58.266674 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:58.266331 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" event={"ID":"6b20c40d-aeae-45ea-8545-2fabe3a2567e","Type":"ContainerDied","Data":"f58f3fcf50046f7cbf251761090c9f62c471aff629ad54706b337f37d2f27e59"} Sep 16 07:39:59.273763 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:59.273734 2580 generic.go:356] "Generic (PLEG): container finished" podID="6b20c40d-aeae-45ea-8545-2fabe3a2567e" containerID="61cba12d08e9b4b928b4e447bb079055cdf5009a0459840f61e299d30fcd9339" exitCode=0 Sep 16 07:39:59.274125 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:39:59.273821 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" event={"ID":"6b20c40d-aeae-45ea-8545-2fabe3a2567e","Type":"ContainerDied","Data":"61cba12d08e9b4b928b4e447bb079055cdf5009a0459840f61e299d30fcd9339"} Sep 16 07:40:00.404521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:00.404497 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:40:00.502748 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:00.502715 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-util\") pod \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " Sep 16 07:40:00.502905 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:00.502774 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-bundle\") pod \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " Sep 16 07:40:00.502905 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:00.502814 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l7jl\" (UniqueName: \"kubernetes.io/projected/6b20c40d-aeae-45ea-8545-2fabe3a2567e-kube-api-access-7l7jl\") pod \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\" (UID: \"6b20c40d-aeae-45ea-8545-2fabe3a2567e\") " Sep 16 07:40:00.503458 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:00.503429 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-bundle" (OuterVolumeSpecName: "bundle") pod "6b20c40d-aeae-45ea-8545-2fabe3a2567e" (UID: "6b20c40d-aeae-45ea-8545-2fabe3a2567e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:40:00.504816 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:00.504788 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b20c40d-aeae-45ea-8545-2fabe3a2567e-kube-api-access-7l7jl" (OuterVolumeSpecName: "kube-api-access-7l7jl") pod "6b20c40d-aeae-45ea-8545-2fabe3a2567e" (UID: "6b20c40d-aeae-45ea-8545-2fabe3a2567e"). InnerVolumeSpecName "kube-api-access-7l7jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:40:00.508348 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:00.508323 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-util" (OuterVolumeSpecName: "util") pod "6b20c40d-aeae-45ea-8545-2fabe3a2567e" (UID: "6b20c40d-aeae-45ea-8545-2fabe3a2567e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:40:00.603846 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:00.603782 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-util\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:00.603846 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:00.603808 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b20c40d-aeae-45ea-8545-2fabe3a2567e-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:00.603846 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:00.603818 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l7jl\" (UniqueName: \"kubernetes.io/projected/6b20c40d-aeae-45ea-8545-2fabe3a2567e-kube-api-access-7l7jl\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:01.285464 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:01.285437 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" Sep 16 07:40:01.285464 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:01.285448 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw2f9l" event={"ID":"6b20c40d-aeae-45ea-8545-2fabe3a2567e","Type":"ContainerDied","Data":"1c1538c33f45cb3428e586af8e57c42e967abc209866be2506947ca79bcfa811"} Sep 16 07:40:01.285666 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:01.285492 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1538c33f45cb3428e586af8e57c42e967abc209866be2506947ca79bcfa811" Sep 16 07:40:07.450897 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.450860 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78b985f7-9kmpr"] Sep 16 07:40:07.452599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.452536 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b20c40d-aeae-45ea-8545-2fabe3a2567e" containerName="util" Sep 16 07:40:07.452599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.452570 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b20c40d-aeae-45ea-8545-2fabe3a2567e" containerName="util" Sep 16 07:40:07.452599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.452590 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b20c40d-aeae-45ea-8545-2fabe3a2567e" containerName="extract" Sep 16 07:40:07.452599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.452600 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b20c40d-aeae-45ea-8545-2fabe3a2567e" containerName="extract" Sep 16 07:40:07.452947 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.452616 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b20c40d-aeae-45ea-8545-2fabe3a2567e" containerName="pull" Sep 16 07:40:07.452947 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.452624 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b20c40d-aeae-45ea-8545-2fabe3a2567e" containerName="pull" Sep 16 07:40:07.452947 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.452798 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b20c40d-aeae-45ea-8545-2fabe3a2567e" containerName="extract" Sep 16 07:40:07.460394 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.460365 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.465611 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.465585 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78b985f7-9kmpr"] Sep 16 07:40:07.563095 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.563060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvmcv\" (UniqueName: \"kubernetes.io/projected/b6d55c77-37f6-4853-9190-a185eb956de3-kube-api-access-rvmcv\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.563281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.563106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-service-ca\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.563281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.563132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-oauth-serving-cert\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.563281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.563159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-trusted-ca-bundle\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.563281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.563207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6d55c77-37f6-4853-9190-a185eb956de3-console-oauth-config\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.563281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.563248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d55c77-37f6-4853-9190-a185eb956de3-console-serving-cert\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.563281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.563264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-console-config\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.663734 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.663706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d55c77-37f6-4853-9190-a185eb956de3-console-serving-cert\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.663734 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.663738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-console-config\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.663921 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.663772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvmcv\" (UniqueName: \"kubernetes.io/projected/b6d55c77-37f6-4853-9190-a185eb956de3-kube-api-access-rvmcv\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.663921 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.663795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-service-ca\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.663921 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.663825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-oauth-serving-cert\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.663921 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.663843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-trusted-ca-bundle\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.663921 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.663872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6d55c77-37f6-4853-9190-a185eb956de3-console-oauth-config\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.664616 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.664592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-service-ca\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.664739 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.664595 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-console-config\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.664739 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.664648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-oauth-serving-cert\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.664956 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.664844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6d55c77-37f6-4853-9190-a185eb956de3-trusted-ca-bundle\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.666293 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.666265 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6d55c77-37f6-4853-9190-a185eb956de3-console-serving-cert\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.666293 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.666291 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6d55c77-37f6-4853-9190-a185eb956de3-console-oauth-config\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.671642 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.671620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvmcv\" (UniqueName: \"kubernetes.io/projected/b6d55c77-37f6-4853-9190-a185eb956de3-kube-api-access-rvmcv\") pod \"console-78b985f7-9kmpr\" (UID: \"b6d55c77-37f6-4853-9190-a185eb956de3\") " pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:07.772743 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:07.772685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:08.043806 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:08.043779 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78b985f7-9kmpr"] Sep 16 07:40:08.046324 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:40:08.046290 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d55c77_37f6_4853_9190_a185eb956de3.slice/crio-0a188b5436561c0b62600f648ef9cb47362cb53a153b62e192c3631ac8883beb WatchSource:0}: Error finding container 0a188b5436561c0b62600f648ef9cb47362cb53a153b62e192c3631ac8883beb: Status 404 returned error can't find the container with id 0a188b5436561c0b62600f648ef9cb47362cb53a153b62e192c3631ac8883beb Sep 16 07:40:08.325005 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:08.324913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b985f7-9kmpr" event={"ID":"b6d55c77-37f6-4853-9190-a185eb956de3","Type":"ContainerStarted","Data":"fde1e7a68700cb68edfd866ee87d5f0ade3deffa0ae102cab810dc7727697d3d"} Sep 16 07:40:08.325005 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:08.324964 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78b985f7-9kmpr" event={"ID":"b6d55c77-37f6-4853-9190-a185eb956de3","Type":"ContainerStarted","Data":"0a188b5436561c0b62600f648ef9cb47362cb53a153b62e192c3631ac8883beb"} Sep 16 07:40:08.342332 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:08.342282 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78b985f7-9kmpr" podStartSLOduration=1.3422662650000001 podStartE2EDuration="1.342266265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:08.340862652 +0000 UTC m=+381.260478613" watchObservedRunningTime="2026-09-16 07:40:08.342266265 +0000 UTC m=+381.261882211" Sep 16 07:40:12.596319 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.596280 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh"] Sep 16 07:40:12.640680 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.640648 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh"] Sep 16 07:40:12.640834 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.640749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.643835 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.643798 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 07:40:12.643970 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.643904 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cljxh\"" Sep 16 07:40:12.644053 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.644037 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 07:40:12.644166 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.644152 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 07:40:12.644273 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.644256 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 07:40:12.807871 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.807837 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71007db5-1d66-4e57-af11-cfc63b4467b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.808037 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.807881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/71007db5-1d66-4e57-af11-cfc63b4467b7-manifests\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.808037 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.807921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71007db5-1d66-4e57-af11-cfc63b4467b7-tmp\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.808037 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.807944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/71007db5-1d66-4e57-af11-cfc63b4467b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.808037 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.807967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5z76\" (UniqueName: \"kubernetes.io/projected/71007db5-1d66-4e57-af11-cfc63b4467b7-kube-api-access-v5z76\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.811519 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.811493 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477"] Sep 16 07:40:12.833889 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.833869 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477"] Sep 16 07:40:12.833996 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.833956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:12.836796 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.836770 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 07:40:12.836892 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.836843 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 07:40:12.837196 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.837181 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l72bd\"" Sep 16 07:40:12.909162 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.909131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71007db5-1d66-4e57-af11-cfc63b4467b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.909328 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.909176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/71007db5-1d66-4e57-af11-cfc63b4467b7-manifests\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.909328 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.909230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2fdd\" (UniqueName: \"kubernetes.io/projected/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-kube-api-access-m2fdd\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:12.909328 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.909280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71007db5-1d66-4e57-af11-cfc63b4467b7-tmp\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.909328 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.909304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/71007db5-1d66-4e57-af11-cfc63b4467b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.909544 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.909330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5z76\" (UniqueName: \"kubernetes.io/projected/71007db5-1d66-4e57-af11-cfc63b4467b7-kube-api-access-v5z76\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.909544 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.909352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:12.909544 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.909436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:12.909696 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.909588 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/71007db5-1d66-4e57-af11-cfc63b4467b7-manifests\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.911750 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.911725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71007db5-1d66-4e57-af11-cfc63b4467b7-tmp\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.911885 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.911857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/71007db5-1d66-4e57-af11-cfc63b4467b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.912059 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.912033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71007db5-1d66-4e57-af11-cfc63b4467b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.918311 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.918284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5z76\" (UniqueName: \"kubernetes.io/projected/71007db5-1d66-4e57-af11-cfc63b4467b7-kube-api-access-v5z76\") pod \"opendatahub-operator-controller-manager-dcb658945-gcqhh\" (UID: \"71007db5-1d66-4e57-af11-cfc63b4467b7\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:12.951098 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:12.951070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:13.010891 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:13.010858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:13.011031 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:13.010914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:13.011031 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:13.010958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2fdd\" (UniqueName: \"kubernetes.io/projected/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-kube-api-access-m2fdd\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:13.011385 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:13.011363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:13.011438 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:13.011364 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:13.019101 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:13.019062 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2fdd\" (UniqueName: \"kubernetes.io/projected/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-kube-api-access-m2fdd\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:13.144740 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:13.144705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:13.255996 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:13.255933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh"] Sep 16 07:40:13.259042 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:40:13.259011 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71007db5_1d66_4e57_af11_cfc63b4467b7.slice/crio-bf0293d19428e1efa619bfcc2ae7cf28b8535027ed3e2c1bc7c423e1adad94ee WatchSource:0}: Error finding container bf0293d19428e1efa619bfcc2ae7cf28b8535027ed3e2c1bc7c423e1adad94ee: Status 404 returned error can't find the container with id bf0293d19428e1efa619bfcc2ae7cf28b8535027ed3e2c1bc7c423e1adad94ee Sep 16 07:40:13.349500 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:13.349468 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" event={"ID":"71007db5-1d66-4e57-af11-cfc63b4467b7","Type":"ContainerStarted","Data":"bf0293d19428e1efa619bfcc2ae7cf28b8535027ed3e2c1bc7c423e1adad94ee"} Sep 16 07:40:13.443942 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:13.443918 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477"] Sep 16 07:40:13.445786 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:40:13.445760 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9a358e_6c81_428d_aa51_4ed71e02a5fb.slice/crio-2aedf30ad3454d9fb13639fb93b4b78dd33ee01493baca98cf6a95f54f5a9052 WatchSource:0}: Error finding container 2aedf30ad3454d9fb13639fb93b4b78dd33ee01493baca98cf6a95f54f5a9052: Status 404 returned error can't find the container with id 2aedf30ad3454d9fb13639fb93b4b78dd33ee01493baca98cf6a95f54f5a9052 Sep 16 07:40:14.359104 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:14.359003 2580 generic.go:356] "Generic (PLEG): container finished" podID="3f9a358e-6c81-428d-aa51-4ed71e02a5fb" containerID="98b15d0ec9d136f95a40f91747a4eee55c6ba54a9304fd82fe06bafaf2f245ad" exitCode=0 Sep 16 07:40:14.359104 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:14.359098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" event={"ID":"3f9a358e-6c81-428d-aa51-4ed71e02a5fb","Type":"ContainerDied","Data":"98b15d0ec9d136f95a40f91747a4eee55c6ba54a9304fd82fe06bafaf2f245ad"} Sep 16 07:40:14.359104 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:14.359163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" event={"ID":"3f9a358e-6c81-428d-aa51-4ed71e02a5fb","Type":"ContainerStarted","Data":"2aedf30ad3454d9fb13639fb93b4b78dd33ee01493baca98cf6a95f54f5a9052"} Sep 16 07:40:17.375865 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:17.375829 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" event={"ID":"71007db5-1d66-4e57-af11-cfc63b4467b7","Type":"ContainerStarted","Data":"07aba536705874711deee902f0c7e5aa423282fa67845890319e09adade1d021"} Sep 16 07:40:17.377968 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:17.377941 2580 generic.go:356] "Generic (PLEG): container finished" podID="3f9a358e-6c81-428d-aa51-4ed71e02a5fb" containerID="6d084ab8c828ba87d4ef835682e58f5bbae4c918e5cc482a42a82d39f045acd4" exitCode=0 Sep 16 07:40:17.378102 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:17.377971 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" event={"ID":"3f9a358e-6c81-428d-aa51-4ed71e02a5fb","Type":"ContainerDied","Data":"6d084ab8c828ba87d4ef835682e58f5bbae4c918e5cc482a42a82d39f045acd4"} Sep 16 07:40:17.773180 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:17.773147 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:17.773351 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:17.773204 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:17.777646 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:17.777621 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:18.384773 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:18.384733 2580 generic.go:356] "Generic (PLEG): container finished" podID="71007db5-1d66-4e57-af11-cfc63b4467b7" containerID="07aba536705874711deee902f0c7e5aa423282fa67845890319e09adade1d021" exitCode=0 Sep 16 07:40:18.385188 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:18.384822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" event={"ID":"71007db5-1d66-4e57-af11-cfc63b4467b7","Type":"ContainerDied","Data":"07aba536705874711deee902f0c7e5aa423282fa67845890319e09adade1d021"} Sep 16 07:40:18.387348 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:18.387323 2580 generic.go:356] "Generic (PLEG): container finished" podID="3f9a358e-6c81-428d-aa51-4ed71e02a5fb" containerID="0fbbdf90fb1158fdb52412d97e9f08cddb82a164f68d27a2e1048dba3761a6a1" exitCode=0 Sep 16 07:40:18.387454 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:18.387406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" event={"ID":"3f9a358e-6c81-428d-aa51-4ed71e02a5fb","Type":"ContainerDied","Data":"0fbbdf90fb1158fdb52412d97e9f08cddb82a164f68d27a2e1048dba3761a6a1"} Sep 16 07:40:18.391442 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:18.391421 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78b985f7-9kmpr" Sep 16 07:40:18.483611 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:18.483580 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-df8c4cb68-lbjqp"] Sep 16 07:40:19.395239 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.395192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" event={"ID":"71007db5-1d66-4e57-af11-cfc63b4467b7","Type":"ContainerStarted","Data":"e420ce8c8e7ccf45b54b9b40ccbc8e1a2661df5e382c447f69969daa6eaf0586"} Sep 16 07:40:19.395650 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.395621 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:19.416958 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.416909 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" podStartSLOduration=2.877652597 podStartE2EDuration="7.416886173s" totalImagesPullingTime="3.44351327s" totalInitContainerRuntime="1.095720306s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:19.414294049 +0000 UTC m=+392.333909997" watchObservedRunningTime="2026-09-16 07:40:19.416886173 +0000 UTC m=+392.336502120" Sep 16 07:40:19.524421 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.524396 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:19.674142 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.674114 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-util\") pod \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " Sep 16 07:40:19.674324 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.674147 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2fdd\" (UniqueName: \"kubernetes.io/projected/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-kube-api-access-m2fdd\") pod \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " Sep 16 07:40:19.674324 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.674256 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-bundle\") pod \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\" (UID: \"3f9a358e-6c81-428d-aa51-4ed71e02a5fb\") " Sep 16 07:40:19.674974 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.674948 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-bundle" (OuterVolumeSpecName: "bundle") pod "3f9a358e-6c81-428d-aa51-4ed71e02a5fb" (UID: "3f9a358e-6c81-428d-aa51-4ed71e02a5fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:40:19.676287 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.676257 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-kube-api-access-m2fdd" (OuterVolumeSpecName: "kube-api-access-m2fdd") pod "3f9a358e-6c81-428d-aa51-4ed71e02a5fb" (UID: "3f9a358e-6c81-428d-aa51-4ed71e02a5fb"). InnerVolumeSpecName "kube-api-access-m2fdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:40:19.678220 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.678178 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-util" (OuterVolumeSpecName: "util") pod "3f9a358e-6c81-428d-aa51-4ed71e02a5fb" (UID: "3f9a358e-6c81-428d-aa51-4ed71e02a5fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:40:19.775642 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.775618 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:19.775642 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.775639 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-util\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:19.775760 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:19.775647 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2fdd\" (UniqueName: \"kubernetes.io/projected/3f9a358e-6c81-428d-aa51-4ed71e02a5fb-kube-api-access-m2fdd\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:20.401888 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:20.401854 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" Sep 16 07:40:20.402287 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:20.401884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qj477" event={"ID":"3f9a358e-6c81-428d-aa51-4ed71e02a5fb","Type":"ContainerDied","Data":"2aedf30ad3454d9fb13639fb93b4b78dd33ee01493baca98cf6a95f54f5a9052"} Sep 16 07:40:20.402287 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:20.401956 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aedf30ad3454d9fb13639fb93b4b78dd33ee01493baca98cf6a95f54f5a9052" Sep 16 07:40:31.409762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:31.409727 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-gcqhh" Sep 16 07:40:34.547358 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.547324 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r"] Sep 16 07:40:34.547732 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.547663 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9a358e-6c81-428d-aa51-4ed71e02a5fb" containerName="util" Sep 16 07:40:34.547732 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.547675 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9a358e-6c81-428d-aa51-4ed71e02a5fb" containerName="util" Sep 16 07:40:34.547732 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.547683 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9a358e-6c81-428d-aa51-4ed71e02a5fb" containerName="pull" Sep 16 07:40:34.547732 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.547688 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9a358e-6c81-428d-aa51-4ed71e02a5fb" containerName="pull" Sep 16 07:40:34.547732 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.547701 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9a358e-6c81-428d-aa51-4ed71e02a5fb" containerName="extract" Sep 16 07:40:34.547732 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.547706 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9a358e-6c81-428d-aa51-4ed71e02a5fb" containerName="extract" Sep 16 07:40:34.547946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.547759 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f9a358e-6c81-428d-aa51-4ed71e02a5fb" containerName="extract" Sep 16 07:40:34.555828 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.555806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.558753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.558727 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 16 07:40:34.558753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.558750 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 16 07:40:34.558909 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.558731 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 07:40:34.559092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.559079 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 16 07:40:34.559239 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.559224 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-27wd5\"" Sep 16 07:40:34.564711 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.564692 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r"] Sep 16 07:40:34.585779 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.585750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2n2\" (UniqueName: \"kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-kube-api-access-4x2n2\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.585871 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.585783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.585871 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.585818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.585871 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.585854 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.585971 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.585884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.585971 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.585924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.586030 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.586004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.687238 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.687193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2n2\" (UniqueName: \"kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-kube-api-access-4x2n2\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.687331 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.687249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.687331 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.687283 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.687331 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.687299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.687331 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.687327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.687522 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.687355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.687522 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.687410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.687993 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.687966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.689594 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.689572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.689784 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.689766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.689879 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.689863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.689932 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.689901 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.696060 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.696038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.696584 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.696564 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2n2\" (UniqueName: \"kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-kube-api-access-4x2n2\") pod \"istiod-openshift-gateway-7b7995d5cd-qxq6r\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:34.866417 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:34.866344 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:35.138047 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:35.138023 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r"] Sep 16 07:40:35.139954 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:40:35.139920 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb37c16d_1b5e_44a6_81ef_d710e9e4140b.slice/crio-6e4b7053e04e5555ea0042bb3e742ce0222dffe89737477ed4d5dbe35f24aeee WatchSource:0}: Error finding container 6e4b7053e04e5555ea0042bb3e742ce0222dffe89737477ed4d5dbe35f24aeee: Status 404 returned error can't find the container with id 6e4b7053e04e5555ea0042bb3e742ce0222dffe89737477ed4d5dbe35f24aeee Sep 16 07:40:35.477384 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:35.477350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" event={"ID":"bb37c16d-1b5e-44a6-81ef-d710e9e4140b","Type":"ContainerStarted","Data":"6e4b7053e04e5555ea0042bb3e742ce0222dffe89737477ed4d5dbe35f24aeee"} Sep 16 07:40:37.766231 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:37.766183 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:40:37.766496 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:37.766263 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:40:38.495848 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:38.495807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" event={"ID":"bb37c16d-1b5e-44a6-81ef-d710e9e4140b","Type":"ContainerStarted","Data":"2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0"} Sep 16 07:40:38.496129 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:38.496062 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:38.497592 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:38.497565 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-qxq6r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 07:40:38.497700 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:38.497612 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" podUID="bb37c16d-1b5e-44a6-81ef-d710e9e4140b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 07:40:38.518776 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:38.518734 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" podStartSLOduration=1.895158999 podStartE2EDuration="4.518722931s" totalImagesPullingTime="2.623563932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:38.516256392 +0000 UTC m=+411.435872339" watchObservedRunningTime="2026-09-16 07:40:38.518722931 +0000 UTC m=+411.438338877" Sep 16 07:40:39.501805 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:39.501776 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:43.503483 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.503420 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-df8c4cb68-lbjqp" podUID="f01e631b-65db-459f-892a-9c32cf1e9e46" containerName="console" containerID="cri-o://4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f" gracePeriod=15 Sep 16 07:40:43.742821 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.742794 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:40:43.762487 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.762423 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-oauth-serving-cert\") pod \"f01e631b-65db-459f-892a-9c32cf1e9e46\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " Sep 16 07:40:43.762487 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.762473 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-trusted-ca-bundle\") pod \"f01e631b-65db-459f-892a-9c32cf1e9e46\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " Sep 16 07:40:43.762656 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.762514 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-oauth-config\") pod \"f01e631b-65db-459f-892a-9c32cf1e9e46\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " Sep 16 07:40:43.762797 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.762769 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kmdr\" (UniqueName: \"kubernetes.io/projected/f01e631b-65db-459f-892a-9c32cf1e9e46-kube-api-access-5kmdr\") pod \"f01e631b-65db-459f-892a-9c32cf1e9e46\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " Sep 16 07:40:43.762926 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.762861 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-serving-cert\") pod \"f01e631b-65db-459f-892a-9c32cf1e9e46\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " Sep 16 07:40:43.762926 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.762909 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-service-ca\") pod \"f01e631b-65db-459f-892a-9c32cf1e9e46\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " Sep 16 07:40:43.763024 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.762857 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f01e631b-65db-459f-892a-9c32cf1e9e46" (UID: "f01e631b-65db-459f-892a-9c32cf1e9e46"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:40:43.763024 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.762957 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-console-config\") pod \"f01e631b-65db-459f-892a-9c32cf1e9e46\" (UID: \"f01e631b-65db-459f-892a-9c32cf1e9e46\") " Sep 16 07:40:43.763685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.762943 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f01e631b-65db-459f-892a-9c32cf1e9e46" (UID: "f01e631b-65db-459f-892a-9c32cf1e9e46"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:40:43.763685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.763302 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-console-config" (OuterVolumeSpecName: "console-config") pod "f01e631b-65db-459f-892a-9c32cf1e9e46" (UID: "f01e631b-65db-459f-892a-9c32cf1e9e46"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:40:43.763685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.763474 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-service-ca" (OuterVolumeSpecName: "service-ca") pod "f01e631b-65db-459f-892a-9c32cf1e9e46" (UID: "f01e631b-65db-459f-892a-9c32cf1e9e46"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:40:43.763685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.763647 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:43.763685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.763664 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-console-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:43.763685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.763676 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-oauth-serving-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:43.763685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.763688 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01e631b-65db-459f-892a-9c32cf1e9e46-trusted-ca-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:43.765530 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.765508 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f01e631b-65db-459f-892a-9c32cf1e9e46" (UID: "f01e631b-65db-459f-892a-9c32cf1e9e46"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:40:43.765530 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.765513 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01e631b-65db-459f-892a-9c32cf1e9e46-kube-api-access-5kmdr" (OuterVolumeSpecName: "kube-api-access-5kmdr") pod "f01e631b-65db-459f-892a-9c32cf1e9e46" (UID: "f01e631b-65db-459f-892a-9c32cf1e9e46"). InnerVolumeSpecName "kube-api-access-5kmdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:40:43.765648 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.765526 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f01e631b-65db-459f-892a-9c32cf1e9e46" (UID: "f01e631b-65db-459f-892a-9c32cf1e9e46"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:40:43.865167 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.865135 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-oauth-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:43.865167 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.865167 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kmdr\" (UniqueName: \"kubernetes.io/projected/f01e631b-65db-459f-892a-9c32cf1e9e46-kube-api-access-5kmdr\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:43.865356 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:43.865181 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f01e631b-65db-459f-892a-9c32cf1e9e46-console-serving-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:44.533493 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:44.533458 2580 generic.go:356] "Generic (PLEG): container finished" podID="f01e631b-65db-459f-892a-9c32cf1e9e46" containerID="4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f" exitCode=0 Sep 16 07:40:44.533949 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:44.533519 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df8c4cb68-lbjqp" Sep 16 07:40:44.533949 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:44.533545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df8c4cb68-lbjqp" event={"ID":"f01e631b-65db-459f-892a-9c32cf1e9e46","Type":"ContainerDied","Data":"4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f"} Sep 16 07:40:44.533949 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:44.533615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df8c4cb68-lbjqp" event={"ID":"f01e631b-65db-459f-892a-9c32cf1e9e46","Type":"ContainerDied","Data":"1747fb1cdc929c4af810814f45dd8a9c8458dac54308caae79a226da38128eb2"} Sep 16 07:40:44.533949 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:44.533635 2580 scope.go:117] "RemoveContainer" containerID="4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f" Sep 16 07:40:44.542489 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:44.542470 2580 scope.go:117] "RemoveContainer" containerID="4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f" Sep 16 07:40:44.542743 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:40:44.542725 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f\": container with ID starting with 4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f not found: ID does not exist" containerID="4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f" Sep 16 07:40:44.542803 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:44.542754 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f"} err="failed to get container status \"4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f\": rpc error: code = NotFound desc = could not find container \"4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f\": container with ID starting with 4f3884ec7f32ce4d283b3dd657abbab3c9a5b21ab8182dadd29c115038bb005f not found: ID does not exist" Sep 16 07:40:44.555080 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:44.555059 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-df8c4cb68-lbjqp"] Sep 16 07:40:44.560847 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:44.560827 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-df8c4cb68-lbjqp"] Sep 16 07:40:45.559588 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:45.559551 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f01e631b-65db-459f-892a-9c32cf1e9e46" path="/var/lib/kubelet/pods/f01e631b-65db-459f-892a-9c32cf1e9e46/volumes" Sep 16 07:40:49.286013 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.285978 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p"] Sep 16 07:40:49.286401 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.286361 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01e631b-65db-459f-892a-9c32cf1e9e46" containerName="console" Sep 16 07:40:49.286401 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.286376 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01e631b-65db-459f-892a-9c32cf1e9e46" containerName="console" Sep 16 07:40:49.286476 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.286446 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f01e631b-65db-459f-892a-9c32cf1e9e46" containerName="console" Sep 16 07:40:49.291800 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.291779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.295651 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.295619 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 16 07:40:49.295651 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.295627 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hprlw\"" Sep 16 07:40:49.295828 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.295687 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 16 07:40:49.295983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.295963 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 16 07:40:49.297931 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.297910 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p"] Sep 16 07:40:49.408409 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.408379 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9q5\" (UniqueName: \"kubernetes.io/projected/e58dec74-9f8d-409a-9c83-69b2150eb575-kube-api-access-mc9q5\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.408546 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.408425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e58dec74-9f8d-409a-9c83-69b2150eb575-metrics-cert\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.408600 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.408545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58dec74-9f8d-409a-9c83-69b2150eb575-cert\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.408600 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.408591 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e58dec74-9f8d-409a-9c83-69b2150eb575-manager-config\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.509566 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.509536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9q5\" (UniqueName: \"kubernetes.io/projected/e58dec74-9f8d-409a-9c83-69b2150eb575-kube-api-access-mc9q5\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.509718 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.509583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e58dec74-9f8d-409a-9c83-69b2150eb575-metrics-cert\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.509718 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.509639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58dec74-9f8d-409a-9c83-69b2150eb575-cert\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.509718 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.509692 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e58dec74-9f8d-409a-9c83-69b2150eb575-manager-config\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.510501 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.510473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e58dec74-9f8d-409a-9c83-69b2150eb575-manager-config\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.512196 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.512173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e58dec74-9f8d-409a-9c83-69b2150eb575-metrics-cert\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.512443 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.512422 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58dec74-9f8d-409a-9c83-69b2150eb575-cert\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.516685 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.516661 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9q5\" (UniqueName: \"kubernetes.io/projected/e58dec74-9f8d-409a-9c83-69b2150eb575-kube-api-access-mc9q5\") pod \"lws-controller-manager-6f78fc697-jw98p\" (UID: \"e58dec74-9f8d-409a-9c83-69b2150eb575\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.602627 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.602564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:49.875389 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:49.875366 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p"] Sep 16 07:40:49.876874 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:40:49.876840 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58dec74_9f8d_409a_9c83_69b2150eb575.slice/crio-8f20c0f4aa939f5bbaab2cb58058c6e30d705a1186c61daf4401af9390ef1347 WatchSource:0}: Error finding container 8f20c0f4aa939f5bbaab2cb58058c6e30d705a1186c61daf4401af9390ef1347: Status 404 returned error can't find the container with id 8f20c0f4aa939f5bbaab2cb58058c6e30d705a1186c61daf4401af9390ef1347 Sep 16 07:40:50.568699 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:50.568667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" event={"ID":"e58dec74-9f8d-409a-9c83-69b2150eb575","Type":"ContainerStarted","Data":"8f20c0f4aa939f5bbaab2cb58058c6e30d705a1186c61daf4401af9390ef1347"} Sep 16 07:40:51.574678 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:51.574645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" event={"ID":"e58dec74-9f8d-409a-9c83-69b2150eb575","Type":"ContainerStarted","Data":"4cf5e248d33807566d45796cfbd59c799217fa83db72e64ffcfc0030ce42a3e3"} Sep 16 07:40:51.575023 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:51.574781 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:40:51.617398 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:51.617349 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" podStartSLOduration=1.0283897 podStartE2EDuration="2.617334724s" totalImagesPullingTime="1.588945024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:51.615823576 +0000 UTC m=+424.535439523" watchObservedRunningTime="2026-09-16 07:40:51.617334724 +0000 UTC m=+424.536950744" Sep 16 07:40:58.490695 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.490657 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r"] Sep 16 07:40:58.491160 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.491005 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" podUID="bb37c16d-1b5e-44a6-81ef-d710e9e4140b" containerName="discovery" containerID="cri-o://2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0" gracePeriod=30 Sep 16 07:40:58.729452 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.729427 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:58.788718 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.788646 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-dns-cert\") pod \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " Sep 16 07:40:58.788718 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.788684 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-local-certs\") pod \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " Sep 16 07:40:58.788718 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.788713 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-token\") pod \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " Sep 16 07:40:58.788974 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.788875 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-cacerts\") pod \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " Sep 16 07:40:58.788974 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.788950 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-ca-configmap\") pod \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " Sep 16 07:40:58.789093 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.789007 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x2n2\" (UniqueName: \"kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-kube-api-access-4x2n2\") pod \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " Sep 16 07:40:58.789093 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.789037 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-kubeconfig\") pod \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\" (UID: \"bb37c16d-1b5e-44a6-81ef-d710e9e4140b\") " Sep 16 07:40:58.789426 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.789387 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "bb37c16d-1b5e-44a6-81ef-d710e9e4140b" (UID: "bb37c16d-1b5e-44a6-81ef-d710e9e4140b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:40:58.791581 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.791546 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb37c16d-1b5e-44a6-81ef-d710e9e4140b" (UID: "bb37c16d-1b5e-44a6-81ef-d710e9e4140b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:40:58.791692 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.791593 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "bb37c16d-1b5e-44a6-81ef-d710e9e4140b" (UID: "bb37c16d-1b5e-44a6-81ef-d710e9e4140b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:40:58.791692 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.791601 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-local-certs" (OuterVolumeSpecName: "local-certs") pod "bb37c16d-1b5e-44a6-81ef-d710e9e4140b" (UID: "bb37c16d-1b5e-44a6-81ef-d710e9e4140b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:40:58.791692 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.791663 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-cacerts" (OuterVolumeSpecName: "cacerts") pod "bb37c16d-1b5e-44a6-81ef-d710e9e4140b" (UID: "bb37c16d-1b5e-44a6-81ef-d710e9e4140b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:40:58.791849 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.791687 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "bb37c16d-1b5e-44a6-81ef-d710e9e4140b" (UID: "bb37c16d-1b5e-44a6-81ef-d710e9e4140b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:40:58.792749 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.792725 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-kube-api-access-4x2n2" (OuterVolumeSpecName: "kube-api-access-4x2n2") pod "bb37c16d-1b5e-44a6-81ef-d710e9e4140b" (UID: "bb37c16d-1b5e-44a6-81ef-d710e9e4140b"). InnerVolumeSpecName "kube-api-access-4x2n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:40:58.890448 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.890424 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-kubeconfig\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:58.890448 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.890445 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-dns-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:58.890567 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.890456 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-local-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:58.890567 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.890465 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:58.890567 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.890472 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-cacerts\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:58.890567 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.890480 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-istio-csr-ca-configmap\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:58.890567 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:58.890488 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x2n2\" (UniqueName: \"kubernetes.io/projected/bb37c16d-1b5e-44a6-81ef-d710e9e4140b-kube-api-access-4x2n2\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:40:59.610843 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:59.610808 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb37c16d-1b5e-44a6-81ef-d710e9e4140b" containerID="2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0" exitCode=0 Sep 16 07:40:59.611256 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:59.610858 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" Sep 16 07:40:59.611256 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:59.610902 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" event={"ID":"bb37c16d-1b5e-44a6-81ef-d710e9e4140b","Type":"ContainerDied","Data":"2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0"} Sep 16 07:40:59.611256 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:59.610955 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r" event={"ID":"bb37c16d-1b5e-44a6-81ef-d710e9e4140b","Type":"ContainerDied","Data":"6e4b7053e04e5555ea0042bb3e742ce0222dffe89737477ed4d5dbe35f24aeee"} Sep 16 07:40:59.611256 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:59.610976 2580 scope.go:117] "RemoveContainer" containerID="2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0" Sep 16 07:40:59.619412 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:59.619394 2580 scope.go:117] "RemoveContainer" containerID="2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0" Sep 16 07:40:59.619688 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:40:59.619665 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0\": container with ID starting with 2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0 not found: ID does not exist" containerID="2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0" Sep 16 07:40:59.619744 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:59.619701 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0"} err="failed to get container status \"2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0\": rpc error: code = NotFound desc = could not find container \"2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0\": container with ID starting with 2101635707f4cb4d0e31b0eb3eb1d8a7d98f90c77caf380144b08627667974d0 not found: ID does not exist" Sep 16 07:40:59.635283 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:59.635256 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r"] Sep 16 07:40:59.635430 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:40:59.635418 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qxq6r"] Sep 16 07:41:01.560331 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:01.560294 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb37c16d-1b5e-44a6-81ef-d710e9e4140b" path="/var/lib/kubelet/pods/bb37c16d-1b5e-44a6-81ef-d710e9e4140b/volumes" Sep 16 07:41:02.581404 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:02.581367 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-jw98p" Sep 16 07:41:48.684339 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.684305 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8"] Sep 16 07:41:48.684709 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.684637 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb37c16d-1b5e-44a6-81ef-d710e9e4140b" containerName="discovery" Sep 16 07:41:48.684709 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.684648 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb37c16d-1b5e-44a6-81ef-d710e9e4140b" containerName="discovery" Sep 16 07:41:48.684781 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.684716 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb37c16d-1b5e-44a6-81ef-d710e9e4140b" containerName="discovery" Sep 16 07:41:48.686785 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.686768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:48.689433 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.689410 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 07:41:48.689529 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.689466 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 07:41:48.690648 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.690633 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l72bd\"" Sep 16 07:41:48.695069 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.695046 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8"] Sep 16 07:41:48.771488 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.771459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:48.771587 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.771494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj49k\" (UniqueName: \"kubernetes.io/projected/46197d69-be28-4576-9ca1-d84ca701b410-kube-api-access-gj49k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:48.771587 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.771544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:48.782279 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.782257 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz"] Sep 16 07:41:48.784517 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.784502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:48.795896 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.795874 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz"] Sep 16 07:41:48.872168 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.872141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:48.872320 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.872199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:48.872320 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.872263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:48.872320 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.872305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj49k\" (UniqueName: \"kubernetes.io/projected/46197d69-be28-4576-9ca1-d84ca701b410-kube-api-access-gj49k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:48.872446 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.872326 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:48.872446 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.872351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkfzq\" (UniqueName: \"kubernetes.io/projected/02c602ae-2d8a-402c-a870-a35cffac0a1d-kube-api-access-xkfzq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:48.872526 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.872511 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:48.872627 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.872606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:48.881760 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.881736 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg"] Sep 16 07:41:48.883547 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.883527 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj49k\" (UniqueName: \"kubernetes.io/projected/46197d69-be28-4576-9ca1-d84ca701b410-kube-api-access-gj49k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:48.884142 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.884129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:48.892983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.892960 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg"] Sep 16 07:41:48.973705 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.973638 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrv6\" (UniqueName: \"kubernetes.io/projected/3a926fe8-1981-46ed-9215-a8b50ef71abe-kube-api-access-wwrv6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:48.973705 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.973699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:48.973855 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.973745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:48.973855 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.973769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:48.973855 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.973790 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:48.973855 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.973809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkfzq\" (UniqueName: \"kubernetes.io/projected/02c602ae-2d8a-402c-a870-a35cffac0a1d-kube-api-access-xkfzq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:48.974068 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.974051 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:48.974104 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.974082 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:48.982570 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.982549 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkfzq\" (UniqueName: \"kubernetes.io/projected/02c602ae-2d8a-402c-a870-a35cffac0a1d-kube-api-access-xkfzq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:48.987741 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.987710 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9"] Sep 16 07:41:48.990315 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.990297 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:48.995976 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:48.995961 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:49.002102 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.002080 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9"] Sep 16 07:41:49.074996 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.074969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:49.075122 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.075011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:49.075122 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.075029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:49.075122 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.075106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:49.075249 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.075148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrv6\" (UniqueName: \"kubernetes.io/projected/3a926fe8-1981-46ed-9215-a8b50ef71abe-kube-api-access-wwrv6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:49.075249 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.075175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhr27\" (UniqueName: \"kubernetes.io/projected/d3835005-e158-4079-be5d-cfe4086692c5-kube-api-access-jhr27\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:49.075426 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.075405 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:49.075543 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.075503 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:49.084598 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.084569 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrv6\" (UniqueName: \"kubernetes.io/projected/3a926fe8-1981-46ed-9215-a8b50ef71abe-kube-api-access-wwrv6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:49.093174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.093156 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:49.175946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.175919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:49.176070 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.175949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:49.176070 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.176010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr27\" (UniqueName: \"kubernetes.io/projected/d3835005-e158-4079-be5d-cfe4086692c5-kube-api-access-jhr27\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:49.176310 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.176292 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:49.176412 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.176388 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:49.183993 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.183973 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhr27\" (UniqueName: \"kubernetes.io/projected/d3835005-e158-4079-be5d-cfe4086692c5-kube-api-access-jhr27\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:49.199906 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.199892 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:49.268497 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.268471 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8"] Sep 16 07:41:49.270693 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:41:49.270663 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46197d69_be28_4576_9ca1_d84ca701b410.slice/crio-bee1074a625bafbcfccfa0360c4a2b5457d80dd6930e35d156cb27b35d3c8f05 WatchSource:0}: Error finding container bee1074a625bafbcfccfa0360c4a2b5457d80dd6930e35d156cb27b35d3c8f05: Status 404 returned error can't find the container with id bee1074a625bafbcfccfa0360c4a2b5457d80dd6930e35d156cb27b35d3c8f05 Sep 16 07:41:49.299105 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.299078 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:49.362940 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.362908 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz"] Sep 16 07:41:49.364829 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:41:49.364797 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c602ae_2d8a_402c_a870_a35cffac0a1d.slice/crio-ff522f3b0ea2c555147dfd16dcd717030f5f014040f6bf36c3961c2473f40909 WatchSource:0}: Error finding container ff522f3b0ea2c555147dfd16dcd717030f5f014040f6bf36c3961c2473f40909: Status 404 returned error can't find the container with id ff522f3b0ea2c555147dfd16dcd717030f5f014040f6bf36c3961c2473f40909 Sep 16 07:41:49.470814 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.470788 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg"] Sep 16 07:41:49.472428 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:41:49.472402 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a926fe8_1981_46ed_9215_a8b50ef71abe.slice/crio-644439ddd55e7b7acaa06584d0bdd691dd695358c9570ae7faf0fb666ae047e7 WatchSource:0}: Error finding container 644439ddd55e7b7acaa06584d0bdd691dd695358c9570ae7faf0fb666ae047e7: Status 404 returned error can't find the container with id 644439ddd55e7b7acaa06584d0bdd691dd695358c9570ae7faf0fb666ae047e7 Sep 16 07:41:49.573802 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.573723 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9"] Sep 16 07:41:49.575355 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:41:49.575330 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3835005_e158_4079_be5d_cfe4086692c5.slice/crio-a8e97791e1706dfaf8059c36173d22bbf2ce4bbf1d79149a6e5d6e3e66bbffb1 WatchSource:0}: Error finding container a8e97791e1706dfaf8059c36173d22bbf2ce4bbf1d79149a6e5d6e3e66bbffb1: Status 404 returned error can't find the container with id a8e97791e1706dfaf8059c36173d22bbf2ce4bbf1d79149a6e5d6e3e66bbffb1 Sep 16 07:41:49.859186 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.859089 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a926fe8-1981-46ed-9215-a8b50ef71abe" containerID="4418e72a3e16b387d416c408e8c944039003a245ed760268359dc3ea4b192dcf" exitCode=0 Sep 16 07:41:49.859186 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.859174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" event={"ID":"3a926fe8-1981-46ed-9215-a8b50ef71abe","Type":"ContainerDied","Data":"4418e72a3e16b387d416c408e8c944039003a245ed760268359dc3ea4b192dcf"} Sep 16 07:41:49.859654 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.859266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" event={"ID":"3a926fe8-1981-46ed-9215-a8b50ef71abe","Type":"ContainerStarted","Data":"644439ddd55e7b7acaa06584d0bdd691dd695358c9570ae7faf0fb666ae047e7"} Sep 16 07:41:49.860929 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.860903 2580 generic.go:356] "Generic (PLEG): container finished" podID="02c602ae-2d8a-402c-a870-a35cffac0a1d" containerID="6aa7e19a63db9e94892f97a0f7291298ac172c907b610ba6a9a91f7a6f3b5dd6" exitCode=0 Sep 16 07:41:49.861030 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.860992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" event={"ID":"02c602ae-2d8a-402c-a870-a35cffac0a1d","Type":"ContainerDied","Data":"6aa7e19a63db9e94892f97a0f7291298ac172c907b610ba6a9a91f7a6f3b5dd6"} Sep 16 07:41:49.861101 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.861039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" event={"ID":"02c602ae-2d8a-402c-a870-a35cffac0a1d","Type":"ContainerStarted","Data":"ff522f3b0ea2c555147dfd16dcd717030f5f014040f6bf36c3961c2473f40909"} Sep 16 07:41:49.862714 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.862689 2580 generic.go:356] "Generic (PLEG): container finished" podID="46197d69-be28-4576-9ca1-d84ca701b410" containerID="37b34f3bcba3e201626839568039b476cf3e2e8792fc5e1299c1cc0bd1cd1799" exitCode=0 Sep 16 07:41:49.862967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.862763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" event={"ID":"46197d69-be28-4576-9ca1-d84ca701b410","Type":"ContainerDied","Data":"37b34f3bcba3e201626839568039b476cf3e2e8792fc5e1299c1cc0bd1cd1799"} Sep 16 07:41:49.862967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.862806 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" event={"ID":"46197d69-be28-4576-9ca1-d84ca701b410","Type":"ContainerStarted","Data":"bee1074a625bafbcfccfa0360c4a2b5457d80dd6930e35d156cb27b35d3c8f05"} Sep 16 07:41:49.864425 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.864406 2580 generic.go:356] "Generic (PLEG): container finished" podID="d3835005-e158-4079-be5d-cfe4086692c5" containerID="887830b04f688bdc16ede9346a90364172dc345e71a6966da37dd934ce4811c8" exitCode=0 Sep 16 07:41:49.864501 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.864480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" event={"ID":"d3835005-e158-4079-be5d-cfe4086692c5","Type":"ContainerDied","Data":"887830b04f688bdc16ede9346a90364172dc345e71a6966da37dd934ce4811c8"} Sep 16 07:41:49.864554 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:49.864507 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" event={"ID":"d3835005-e158-4079-be5d-cfe4086692c5","Type":"ContainerStarted","Data":"a8e97791e1706dfaf8059c36173d22bbf2ce4bbf1d79149a6e5d6e3e66bbffb1"} Sep 16 07:41:50.872046 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:50.871958 2580 generic.go:356] "Generic (PLEG): container finished" podID="46197d69-be28-4576-9ca1-d84ca701b410" containerID="ce5d0e1a7e17a7edcb5fd90bd273f53b89911808bac1c74bd6de368d699bb84e" exitCode=0 Sep 16 07:41:50.872046 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:50.871994 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" event={"ID":"46197d69-be28-4576-9ca1-d84ca701b410","Type":"ContainerDied","Data":"ce5d0e1a7e17a7edcb5fd90bd273f53b89911808bac1c74bd6de368d699bb84e"} Sep 16 07:41:51.878254 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:51.878223 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a926fe8-1981-46ed-9215-a8b50ef71abe" containerID="8aa1a60fc04b4546a7b48a15acfe7dcce814f315b07bb5cd829d5e181e9c9eb9" exitCode=0 Sep 16 07:41:51.878656 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:51.878302 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" event={"ID":"3a926fe8-1981-46ed-9215-a8b50ef71abe","Type":"ContainerDied","Data":"8aa1a60fc04b4546a7b48a15acfe7dcce814f315b07bb5cd829d5e181e9c9eb9"} Sep 16 07:41:51.880361 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:51.880338 2580 generic.go:356] "Generic (PLEG): container finished" podID="02c602ae-2d8a-402c-a870-a35cffac0a1d" containerID="cbadb5afa02235bba6b7183ac9db4f59c5701b0002e51928130dae68c7d0b58c" exitCode=0 Sep 16 07:41:51.880489 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:51.880412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" event={"ID":"02c602ae-2d8a-402c-a870-a35cffac0a1d","Type":"ContainerDied","Data":"cbadb5afa02235bba6b7183ac9db4f59c5701b0002e51928130dae68c7d0b58c"} Sep 16 07:41:51.882643 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:51.882623 2580 generic.go:356] "Generic (PLEG): container finished" podID="46197d69-be28-4576-9ca1-d84ca701b410" containerID="82145b007204f3ab928590b3c173f7f2910ea051ccfc4958470bf4b659750e29" exitCode=0 Sep 16 07:41:51.882730 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:51.882702 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" event={"ID":"46197d69-be28-4576-9ca1-d84ca701b410","Type":"ContainerDied","Data":"82145b007204f3ab928590b3c173f7f2910ea051ccfc4958470bf4b659750e29"} Sep 16 07:41:52.895229 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:52.890173 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a926fe8-1981-46ed-9215-a8b50ef71abe" containerID="3cb4458d9d7b847b9d17a30f29aa38a9ee04b73c4c7e6cfdfd3e1835bd3971f5" exitCode=0 Sep 16 07:41:52.895229 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:52.890323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" event={"ID":"3a926fe8-1981-46ed-9215-a8b50ef71abe","Type":"ContainerDied","Data":"3cb4458d9d7b847b9d17a30f29aa38a9ee04b73c4c7e6cfdfd3e1835bd3971f5"} Sep 16 07:41:52.898280 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:52.898255 2580 generic.go:356] "Generic (PLEG): container finished" podID="02c602ae-2d8a-402c-a870-a35cffac0a1d" containerID="56cd1bd51db58562d0205da66d9e539cdd3401b2a7f2ce8b8cfd68a938233372" exitCode=0 Sep 16 07:41:52.898418 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:52.898337 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" event={"ID":"02c602ae-2d8a-402c-a870-a35cffac0a1d","Type":"ContainerDied","Data":"56cd1bd51db58562d0205da66d9e539cdd3401b2a7f2ce8b8cfd68a938233372"} Sep 16 07:41:53.026412 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.026387 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:53.115191 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.115163 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-bundle\") pod \"46197d69-be28-4576-9ca1-d84ca701b410\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " Sep 16 07:41:53.115336 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.115200 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj49k\" (UniqueName: \"kubernetes.io/projected/46197d69-be28-4576-9ca1-d84ca701b410-kube-api-access-gj49k\") pod \"46197d69-be28-4576-9ca1-d84ca701b410\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " Sep 16 07:41:53.115336 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.115265 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-util\") pod \"46197d69-be28-4576-9ca1-d84ca701b410\" (UID: \"46197d69-be28-4576-9ca1-d84ca701b410\") " Sep 16 07:41:53.115718 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.115692 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-bundle" (OuterVolumeSpecName: "bundle") pod "46197d69-be28-4576-9ca1-d84ca701b410" (UID: "46197d69-be28-4576-9ca1-d84ca701b410"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:41:53.117490 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.117465 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46197d69-be28-4576-9ca1-d84ca701b410-kube-api-access-gj49k" (OuterVolumeSpecName: "kube-api-access-gj49k") pod "46197d69-be28-4576-9ca1-d84ca701b410" (UID: "46197d69-be28-4576-9ca1-d84ca701b410"). InnerVolumeSpecName "kube-api-access-gj49k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:41:53.120978 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.120959 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-util" (OuterVolumeSpecName: "util") pod "46197d69-be28-4576-9ca1-d84ca701b410" (UID: "46197d69-be28-4576-9ca1-d84ca701b410"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:41:53.215960 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.215936 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj49k\" (UniqueName: \"kubernetes.io/projected/46197d69-be28-4576-9ca1-d84ca701b410-kube-api-access-gj49k\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:53.215960 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.215957 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-util\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:53.216120 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.215968 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46197d69-be28-4576-9ca1-d84ca701b410-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:53.905666 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.905639 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" Sep 16 07:41:53.906017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.905670 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2bv8" event={"ID":"46197d69-be28-4576-9ca1-d84ca701b410","Type":"ContainerDied","Data":"bee1074a625bafbcfccfa0360c4a2b5457d80dd6930e35d156cb27b35d3c8f05"} Sep 16 07:41:53.906017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:53.905714 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bee1074a625bafbcfccfa0360c4a2b5457d80dd6930e35d156cb27b35d3c8f05" Sep 16 07:41:54.035711 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.035688 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:54.066361 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.066340 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:54.122736 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.122712 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-bundle\") pod \"02c602ae-2d8a-402c-a870-a35cffac0a1d\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " Sep 16 07:41:54.122859 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.122768 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkfzq\" (UniqueName: \"kubernetes.io/projected/02c602ae-2d8a-402c-a870-a35cffac0a1d-kube-api-access-xkfzq\") pod \"02c602ae-2d8a-402c-a870-a35cffac0a1d\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " Sep 16 07:41:54.122859 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.122791 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwrv6\" (UniqueName: \"kubernetes.io/projected/3a926fe8-1981-46ed-9215-a8b50ef71abe-kube-api-access-wwrv6\") pod \"3a926fe8-1981-46ed-9215-a8b50ef71abe\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " Sep 16 07:41:54.122859 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.122806 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-util\") pod \"3a926fe8-1981-46ed-9215-a8b50ef71abe\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " Sep 16 07:41:54.122859 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.122827 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-util\") pod \"02c602ae-2d8a-402c-a870-a35cffac0a1d\" (UID: \"02c602ae-2d8a-402c-a870-a35cffac0a1d\") " Sep 16 07:41:54.123077 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.122863 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-bundle\") pod \"3a926fe8-1981-46ed-9215-a8b50ef71abe\" (UID: \"3a926fe8-1981-46ed-9215-a8b50ef71abe\") " Sep 16 07:41:54.123434 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.123408 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-bundle" (OuterVolumeSpecName: "bundle") pod "3a926fe8-1981-46ed-9215-a8b50ef71abe" (UID: "3a926fe8-1981-46ed-9215-a8b50ef71abe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:41:54.123608 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.123586 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-bundle" (OuterVolumeSpecName: "bundle") pod "02c602ae-2d8a-402c-a870-a35cffac0a1d" (UID: "02c602ae-2d8a-402c-a870-a35cffac0a1d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:41:54.124980 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.124948 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c602ae-2d8a-402c-a870-a35cffac0a1d-kube-api-access-xkfzq" (OuterVolumeSpecName: "kube-api-access-xkfzq") pod "02c602ae-2d8a-402c-a870-a35cffac0a1d" (UID: "02c602ae-2d8a-402c-a870-a35cffac0a1d"). InnerVolumeSpecName "kube-api-access-xkfzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:41:54.125363 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.125344 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a926fe8-1981-46ed-9215-a8b50ef71abe-kube-api-access-wwrv6" (OuterVolumeSpecName: "kube-api-access-wwrv6") pod "3a926fe8-1981-46ed-9215-a8b50ef71abe" (UID: "3a926fe8-1981-46ed-9215-a8b50ef71abe"). InnerVolumeSpecName "kube-api-access-wwrv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:41:54.128375 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.128356 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-util" (OuterVolumeSpecName: "util") pod "3a926fe8-1981-46ed-9215-a8b50ef71abe" (UID: "3a926fe8-1981-46ed-9215-a8b50ef71abe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:41:54.128927 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.128905 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-util" (OuterVolumeSpecName: "util") pod "02c602ae-2d8a-402c-a870-a35cffac0a1d" (UID: "02c602ae-2d8a-402c-a870-a35cffac0a1d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:41:54.223480 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.223419 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwrv6\" (UniqueName: \"kubernetes.io/projected/3a926fe8-1981-46ed-9215-a8b50ef71abe-kube-api-access-wwrv6\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:54.223480 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.223445 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-util\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:54.223480 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.223459 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-util\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:54.223480 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.223471 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a926fe8-1981-46ed-9215-a8b50ef71abe-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:54.223659 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.223482 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c602ae-2d8a-402c-a870-a35cffac0a1d-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:54.223659 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.223494 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkfzq\" (UniqueName: \"kubernetes.io/projected/02c602ae-2d8a-402c-a870-a35cffac0a1d-kube-api-access-xkfzq\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:54.915395 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.915364 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" Sep 16 07:41:54.915873 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.915363 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62vrmg" event={"ID":"3a926fe8-1981-46ed-9215-a8b50ef71abe","Type":"ContainerDied","Data":"644439ddd55e7b7acaa06584d0bdd691dd695358c9570ae7faf0fb666ae047e7"} Sep 16 07:41:54.915873 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.915500 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644439ddd55e7b7acaa06584d0bdd691dd695358c9570ae7faf0fb666ae047e7" Sep 16 07:41:54.917642 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.917618 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" Sep 16 07:41:54.917779 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.917623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anvfgz" event={"ID":"02c602ae-2d8a-402c-a870-a35cffac0a1d","Type":"ContainerDied","Data":"ff522f3b0ea2c555147dfd16dcd717030f5f014040f6bf36c3961c2473f40909"} Sep 16 07:41:54.917779 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:54.917731 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff522f3b0ea2c555147dfd16dcd717030f5f014040f6bf36c3961c2473f40909" Sep 16 07:41:56.929688 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:56.929654 2580 generic.go:356] "Generic (PLEG): container finished" podID="d3835005-e158-4079-be5d-cfe4086692c5" containerID="978174b7a2186299659720a9e7b98a73071597966929103fef070d115393fbbb" exitCode=0 Sep 16 07:41:56.930086 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:56.929728 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" event={"ID":"d3835005-e158-4079-be5d-cfe4086692c5","Type":"ContainerDied","Data":"978174b7a2186299659720a9e7b98a73071597966929103fef070d115393fbbb"} Sep 16 07:41:57.936837 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:57.936805 2580 generic.go:356] "Generic (PLEG): container finished" podID="d3835005-e158-4079-be5d-cfe4086692c5" containerID="275673a2dcca0446feb12a94941064f80cd78793f72033754bc2b3bfd6b79887" exitCode=0 Sep 16 07:41:57.937248 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:57.936895 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" event={"ID":"d3835005-e158-4079-be5d-cfe4086692c5","Type":"ContainerDied","Data":"275673a2dcca0446feb12a94941064f80cd78793f72033754bc2b3bfd6b79887"} Sep 16 07:41:59.068516 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.068494 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:59.158806 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.158782 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhr27\" (UniqueName: \"kubernetes.io/projected/d3835005-e158-4079-be5d-cfe4086692c5-kube-api-access-jhr27\") pod \"d3835005-e158-4079-be5d-cfe4086692c5\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " Sep 16 07:41:59.158948 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.158844 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-util\") pod \"d3835005-e158-4079-be5d-cfe4086692c5\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " Sep 16 07:41:59.158948 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.158898 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-bundle\") pod \"d3835005-e158-4079-be5d-cfe4086692c5\" (UID: \"d3835005-e158-4079-be5d-cfe4086692c5\") " Sep 16 07:41:59.159397 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.159369 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-bundle" (OuterVolumeSpecName: "bundle") pod "d3835005-e158-4079-be5d-cfe4086692c5" (UID: "d3835005-e158-4079-be5d-cfe4086692c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:41:59.160788 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.160768 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3835005-e158-4079-be5d-cfe4086692c5-kube-api-access-jhr27" (OuterVolumeSpecName: "kube-api-access-jhr27") pod "d3835005-e158-4079-be5d-cfe4086692c5" (UID: "d3835005-e158-4079-be5d-cfe4086692c5"). InnerVolumeSpecName "kube-api-access-jhr27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:41:59.163275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.163241 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-util" (OuterVolumeSpecName: "util") pod "d3835005-e158-4079-be5d-cfe4086692c5" (UID: "d3835005-e158-4079-be5d-cfe4086692c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:41:59.260079 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.260022 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:59.260079 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.260047 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhr27\" (UniqueName: \"kubernetes.io/projected/d3835005-e158-4079-be5d-cfe4086692c5-kube-api-access-jhr27\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:59.260079 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.260057 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3835005-e158-4079-be5d-cfe4086692c5-util\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:41:59.949323 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.949297 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" Sep 16 07:41:59.949471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.949297 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct69h9" event={"ID":"d3835005-e158-4079-be5d-cfe4086692c5","Type":"ContainerDied","Data":"a8e97791e1706dfaf8059c36173d22bbf2ce4bbf1d79149a6e5d6e3e66bbffb1"} Sep 16 07:41:59.949471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:41:59.949430 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8e97791e1706dfaf8059c36173d22bbf2ce4bbf1d79149a6e5d6e3e66bbffb1" Sep 16 07:42:07.163282 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163237 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll"] Sep 16 07:42:07.163806 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163789 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3835005-e158-4079-be5d-cfe4086692c5" containerName="util" Sep 16 07:42:07.163852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163811 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3835005-e158-4079-be5d-cfe4086692c5" containerName="util" Sep 16 07:42:07.163852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163834 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46197d69-be28-4576-9ca1-d84ca701b410" containerName="util" Sep 16 07:42:07.163852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163846 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="46197d69-be28-4576-9ca1-d84ca701b410" containerName="util" Sep 16 07:42:07.163946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163857 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c602ae-2d8a-402c-a870-a35cffac0a1d" containerName="pull" Sep 16 07:42:07.163946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163867 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c602ae-2d8a-402c-a870-a35cffac0a1d" containerName="pull" Sep 16 07:42:07.163946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163880 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c602ae-2d8a-402c-a870-a35cffac0a1d" containerName="util" Sep 16 07:42:07.163946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163889 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c602ae-2d8a-402c-a870-a35cffac0a1d" containerName="util" Sep 16 07:42:07.163946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163904 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a926fe8-1981-46ed-9215-a8b50ef71abe" containerName="util" Sep 16 07:42:07.163946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163912 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a926fe8-1981-46ed-9215-a8b50ef71abe" containerName="util" Sep 16 07:42:07.163946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163922 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a926fe8-1981-46ed-9215-a8b50ef71abe" containerName="pull" Sep 16 07:42:07.163946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163931 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a926fe8-1981-46ed-9215-a8b50ef71abe" containerName="pull" Sep 16 07:42:07.163946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163943 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3835005-e158-4079-be5d-cfe4086692c5" containerName="pull" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163953 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3835005-e158-4079-be5d-cfe4086692c5" containerName="pull" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163967 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46197d69-be28-4576-9ca1-d84ca701b410" containerName="pull" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163977 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="46197d69-be28-4576-9ca1-d84ca701b410" containerName="pull" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.163990 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46197d69-be28-4576-9ca1-d84ca701b410" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164000 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="46197d69-be28-4576-9ca1-d84ca701b410" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164015 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3835005-e158-4079-be5d-cfe4086692c5" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164025 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3835005-e158-4079-be5d-cfe4086692c5" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164038 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a926fe8-1981-46ed-9215-a8b50ef71abe" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164048 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a926fe8-1981-46ed-9215-a8b50ef71abe" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164060 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c602ae-2d8a-402c-a870-a35cffac0a1d" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164069 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c602ae-2d8a-402c-a870-a35cffac0a1d" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164155 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3835005-e158-4079-be5d-cfe4086692c5" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164174 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="46197d69-be28-4576-9ca1-d84ca701b410" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164184 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a926fe8-1981-46ed-9215-a8b50ef71abe" containerName="extract" Sep 16 07:42:07.164241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.164195 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c602ae-2d8a-402c-a870-a35cffac0a1d" containerName="extract" Sep 16 07:42:07.167940 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.167916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll" Sep 16 07:42:07.171349 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.171326 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 16 07:42:07.171728 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.171708 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 07:42:07.172014 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.171971 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mjqmk\"" Sep 16 07:42:07.172539 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.172524 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 07:42:07.180122 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.180093 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll"] Sep 16 07:42:07.223741 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.223715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdnpd\" (UniqueName: \"kubernetes.io/projected/27748d8c-eb94-4e23-977d-62c8314595f0-kube-api-access-fdnpd\") pod \"dns-operator-controller-manager-7b84cd7c9c-4vfll\" (UID: \"27748d8c-eb94-4e23-977d-62c8314595f0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll" Sep 16 07:42:07.325027 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.324992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdnpd\" (UniqueName: \"kubernetes.io/projected/27748d8c-eb94-4e23-977d-62c8314595f0-kube-api-access-fdnpd\") pod \"dns-operator-controller-manager-7b84cd7c9c-4vfll\" (UID: \"27748d8c-eb94-4e23-977d-62c8314595f0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll" Sep 16 07:42:07.337318 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.337295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdnpd\" (UniqueName: \"kubernetes.io/projected/27748d8c-eb94-4e23-977d-62c8314595f0-kube-api-access-fdnpd\") pod \"dns-operator-controller-manager-7b84cd7c9c-4vfll\" (UID: \"27748d8c-eb94-4e23-977d-62c8314595f0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll" Sep 16 07:42:07.484496 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.484436 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll" Sep 16 07:42:07.761654 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.761627 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll"] Sep 16 07:42:07.763565 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:42:07.763541 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27748d8c_eb94_4e23_977d_62c8314595f0.slice/crio-3d8a521659cb4fa5ff7b0e8592e461bfca9eadee820a11354a23e66677fb7264 WatchSource:0}: Error finding container 3d8a521659cb4fa5ff7b0e8592e461bfca9eadee820a11354a23e66677fb7264: Status 404 returned error can't find the container with id 3d8a521659cb4fa5ff7b0e8592e461bfca9eadee820a11354a23e66677fb7264 Sep 16 07:42:07.992527 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:07.992487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll" event={"ID":"27748d8c-eb94-4e23-977d-62c8314595f0","Type":"ContainerStarted","Data":"3d8a521659cb4fa5ff7b0e8592e461bfca9eadee820a11354a23e66677fb7264"} Sep 16 07:42:09.062183 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:09.062148 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs"] Sep 16 07:42:09.065773 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:09.065752 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:09.068883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:09.068857 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-q87hk\"" Sep 16 07:42:09.079291 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:09.079255 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs"] Sep 16 07:42:09.140855 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:09.140823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgsc9\" (UniqueName: \"kubernetes.io/projected/7ee4812a-3e41-47eb-8c98-7c7f5ab175ac-kube-api-access-lgsc9\") pod \"limitador-operator-controller-manager-74495c69f8-6m5zs\" (UID: \"7ee4812a-3e41-47eb-8c98-7c7f5ab175ac\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:09.241671 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:09.241628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgsc9\" (UniqueName: \"kubernetes.io/projected/7ee4812a-3e41-47eb-8c98-7c7f5ab175ac-kube-api-access-lgsc9\") pod \"limitador-operator-controller-manager-74495c69f8-6m5zs\" (UID: \"7ee4812a-3e41-47eb-8c98-7c7f5ab175ac\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:09.259647 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:09.259621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgsc9\" (UniqueName: \"kubernetes.io/projected/7ee4812a-3e41-47eb-8c98-7c7f5ab175ac-kube-api-access-lgsc9\") pod \"limitador-operator-controller-manager-74495c69f8-6m5zs\" (UID: \"7ee4812a-3e41-47eb-8c98-7c7f5ab175ac\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:09.379505 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:09.379436 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:10.445314 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:10.445290 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs"] Sep 16 07:42:10.446698 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:42:10.446673 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee4812a_3e41_47eb_8c98_7c7f5ab175ac.slice/crio-09a939b403f01db93ec6867816ef32c8d6506b8983fbab8c54f2593600832e41 WatchSource:0}: Error finding container 09a939b403f01db93ec6867816ef32c8d6506b8983fbab8c54f2593600832e41: Status 404 returned error can't find the container with id 09a939b403f01db93ec6867816ef32c8d6506b8983fbab8c54f2593600832e41 Sep 16 07:42:11.013475 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:11.013437 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll" event={"ID":"27748d8c-eb94-4e23-977d-62c8314595f0","Type":"ContainerStarted","Data":"a2c89d1fa47a0a42d33189385b95b73e3ff754757440f85c82efeb306a224890"} Sep 16 07:42:11.013669 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:11.013551 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll" Sep 16 07:42:11.014956 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:11.014929 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" event={"ID":"7ee4812a-3e41-47eb-8c98-7c7f5ab175ac","Type":"ContainerStarted","Data":"09a939b403f01db93ec6867816ef32c8d6506b8983fbab8c54f2593600832e41"} Sep 16 07:42:11.065769 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:11.065724 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll" podStartSLOduration=1.588581467 podStartE2EDuration="4.065711685s" totalImagesPullingTime="2.477130218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:42:11.063744652 +0000 UTC m=+503.983360598" watchObservedRunningTime="2026-09-16 07:42:11.065711685 +0000 UTC m=+503.985327630" Sep 16 07:42:12.022802 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:12.022765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" event={"ID":"7ee4812a-3e41-47eb-8c98-7c7f5ab175ac","Type":"ContainerStarted","Data":"915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9"} Sep 16 07:42:12.043456 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:12.043410 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" podStartSLOduration=1.573925078 podStartE2EDuration="3.043394998s" totalImagesPullingTime="1.46946992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:42:12.042160052 +0000 UTC m=+504.961776000" watchObservedRunningTime="2026-09-16 07:42:12.043394998 +0000 UTC m=+504.963010934" Sep 16 07:42:13.029615 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.029582 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:13.567315 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.567284 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq"] Sep 16 07:42:13.570664 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.570647 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:13.573809 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.573778 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xbvtl\"" Sep 16 07:42:13.592306 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.592276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq"] Sep 16 07:42:13.688668 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.688639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5930c8e5-111a-4cab-bb13-164ba8b04f95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" (UID: \"5930c8e5-111a-4cab-bb13-164ba8b04f95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:13.688811 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.688699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9gx4\" (UniqueName: \"kubernetes.io/projected/5930c8e5-111a-4cab-bb13-164ba8b04f95-kube-api-access-k9gx4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" (UID: \"5930c8e5-111a-4cab-bb13-164ba8b04f95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:13.789567 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.789536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9gx4\" (UniqueName: \"kubernetes.io/projected/5930c8e5-111a-4cab-bb13-164ba8b04f95-kube-api-access-k9gx4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" (UID: \"5930c8e5-111a-4cab-bb13-164ba8b04f95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:13.789712 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.789603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5930c8e5-111a-4cab-bb13-164ba8b04f95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" (UID: \"5930c8e5-111a-4cab-bb13-164ba8b04f95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:13.789921 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.789904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5930c8e5-111a-4cab-bb13-164ba8b04f95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" (UID: \"5930c8e5-111a-4cab-bb13-164ba8b04f95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:13.799400 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.799372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9gx4\" (UniqueName: \"kubernetes.io/projected/5930c8e5-111a-4cab-bb13-164ba8b04f95-kube-api-access-k9gx4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" (UID: \"5930c8e5-111a-4cab-bb13-164ba8b04f95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:13.880701 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:13.880644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:14.171044 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:14.171014 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq"] Sep 16 07:42:14.172964 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:42:14.172938 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5930c8e5_111a_4cab_bb13_164ba8b04f95.slice/crio-0cfa372f326bfe6b67b88cfddb716f7fa6cf3192883684e39d1a280d67861704 WatchSource:0}: Error finding container 0cfa372f326bfe6b67b88cfddb716f7fa6cf3192883684e39d1a280d67861704: Status 404 returned error can't find the container with id 0cfa372f326bfe6b67b88cfddb716f7fa6cf3192883684e39d1a280d67861704 Sep 16 07:42:15.041281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:15.041241 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" event={"ID":"5930c8e5-111a-4cab-bb13-164ba8b04f95","Type":"ContainerStarted","Data":"0cfa372f326bfe6b67b88cfddb716f7fa6cf3192883684e39d1a280d67861704"} Sep 16 07:42:19.068862 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:19.068821 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" event={"ID":"5930c8e5-111a-4cab-bb13-164ba8b04f95","Type":"ContainerStarted","Data":"5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60"} Sep 16 07:42:19.069261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:19.068977 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:19.101756 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:19.101711 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" podStartSLOduration=2.167586303 podStartE2EDuration="6.101696616s" totalImagesPullingTime="3.934110313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:42:19.09978476 +0000 UTC m=+512.019400705" watchObservedRunningTime="2026-09-16 07:42:19.101696616 +0000 UTC m=+512.021312561" Sep 16 07:42:22.025343 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:22.025311 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4vfll" Sep 16 07:42:24.036945 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:24.036909 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:30.076858 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:30.076810 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:32.233568 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.233532 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq"] Sep 16 07:42:32.233948 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.233717 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" containerName="manager" containerID="cri-o://5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60" gracePeriod=2 Sep 16 07:42:32.238934 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.238902 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq"] Sep 16 07:42:32.254349 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.254322 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs"] Sep 16 07:42:32.254530 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.254509 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" containerName="manager" containerID="cri-o://915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9" gracePeriod=2 Sep 16 07:42:32.269558 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.269529 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs"] Sep 16 07:42:32.289993 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:32.289952 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"7ee4812a-3e41-47eb-8c98-7c7f5ab175ac\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-6m5zs\": pods \"limitador-operator-controller-manager-74495c69f8-6m5zs\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:32.290623 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.290596 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx"] Sep 16 07:42:32.291054 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.291030 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" containerName="manager" Sep 16 07:42:32.291054 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.291054 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" containerName="manager" Sep 16 07:42:32.291235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.291088 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" containerName="manager" Sep 16 07:42:32.291235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.291096 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" containerName="manager" Sep 16 07:42:32.291235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.291186 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" containerName="manager" Sep 16 07:42:32.291235 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.291204 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" containerName="manager" Sep 16 07:42:32.293302 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:32.293276 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:32.294471 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.294449 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8"] Sep 16 07:42:32.294584 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.294563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx" Sep 16 07:42:32.296887 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:32.296860 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6m5zs\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:32.298116 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.298096 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:32.311568 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.311543 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx"] Sep 16 07:42:32.317470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.317447 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8"] Sep 16 07:42:32.334363 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:32.334336 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:32.336593 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:32.336570 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6m5zs\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:32.338672 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:32.338642 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:32.446455 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.446425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6dm\" (UniqueName: \"kubernetes.io/projected/7914ba72-02eb-4977-be9c-128174d504ba-kube-api-access-gm6dm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4pvc8\" (UID: \"7914ba72-02eb-4977-be9c-128174d504ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:32.446596 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.446470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6drj6\" (UniqueName: \"kubernetes.io/projected/cb7a369c-acbf-4bbd-96b8-3a5e3df15c58-kube-api-access-6drj6\") pod \"limitador-operator-controller-manager-74495c69f8-8q5lx\" (UID: \"cb7a369c-acbf-4bbd-96b8-3a5e3df15c58\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx" Sep 16 07:42:32.446596 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.446496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7914ba72-02eb-4977-be9c-128174d504ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4pvc8\" (UID: \"7914ba72-02eb-4977-be9c-128174d504ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:32.497184 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.497165 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:32.499949 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:32.499924 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6m5zs\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:32.500044 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.500009 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:32.502349 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:32.502327 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:32.504567 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:32.504541 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6m5zs\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:32.506617 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:32.506594 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:32.546970 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.546940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6dm\" (UniqueName: \"kubernetes.io/projected/7914ba72-02eb-4977-be9c-128174d504ba-kube-api-access-gm6dm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4pvc8\" (UID: \"7914ba72-02eb-4977-be9c-128174d504ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:32.547085 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.546977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6drj6\" (UniqueName: \"kubernetes.io/projected/cb7a369c-acbf-4bbd-96b8-3a5e3df15c58-kube-api-access-6drj6\") pod \"limitador-operator-controller-manager-74495c69f8-8q5lx\" (UID: \"cb7a369c-acbf-4bbd-96b8-3a5e3df15c58\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx" Sep 16 07:42:32.547085 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.547002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7914ba72-02eb-4977-be9c-128174d504ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4pvc8\" (UID: \"7914ba72-02eb-4977-be9c-128174d504ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:32.547389 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.547371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7914ba72-02eb-4977-be9c-128174d504ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4pvc8\" (UID: \"7914ba72-02eb-4977-be9c-128174d504ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:32.565098 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.565071 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6drj6\" (UniqueName: \"kubernetes.io/projected/cb7a369c-acbf-4bbd-96b8-3a5e3df15c58-kube-api-access-6drj6\") pod \"limitador-operator-controller-manager-74495c69f8-8q5lx\" (UID: \"cb7a369c-acbf-4bbd-96b8-3a5e3df15c58\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx" Sep 16 07:42:32.565179 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.565128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6dm\" (UniqueName: \"kubernetes.io/projected/7914ba72-02eb-4977-be9c-128174d504ba-kube-api-access-gm6dm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4pvc8\" (UID: \"7914ba72-02eb-4977-be9c-128174d504ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:32.648289 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.648265 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5930c8e5-111a-4cab-bb13-164ba8b04f95-extensions-socket-volume\") pod \"5930c8e5-111a-4cab-bb13-164ba8b04f95\" (UID: \"5930c8e5-111a-4cab-bb13-164ba8b04f95\") " Sep 16 07:42:32.648362 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.648302 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgsc9\" (UniqueName: \"kubernetes.io/projected/7ee4812a-3e41-47eb-8c98-7c7f5ab175ac-kube-api-access-lgsc9\") pod \"7ee4812a-3e41-47eb-8c98-7c7f5ab175ac\" (UID: \"7ee4812a-3e41-47eb-8c98-7c7f5ab175ac\") " Sep 16 07:42:32.648408 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.648363 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9gx4\" (UniqueName: \"kubernetes.io/projected/5930c8e5-111a-4cab-bb13-164ba8b04f95-kube-api-access-k9gx4\") pod \"5930c8e5-111a-4cab-bb13-164ba8b04f95\" (UID: \"5930c8e5-111a-4cab-bb13-164ba8b04f95\") " Sep 16 07:42:32.648798 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.648773 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5930c8e5-111a-4cab-bb13-164ba8b04f95-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5930c8e5-111a-4cab-bb13-164ba8b04f95" (UID: "5930c8e5-111a-4cab-bb13-164ba8b04f95"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:42:32.650240 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.650205 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee4812a-3e41-47eb-8c98-7c7f5ab175ac-kube-api-access-lgsc9" (OuterVolumeSpecName: "kube-api-access-lgsc9") pod "7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" (UID: "7ee4812a-3e41-47eb-8c98-7c7f5ab175ac"). InnerVolumeSpecName "kube-api-access-lgsc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:42:32.650354 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.650337 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5930c8e5-111a-4cab-bb13-164ba8b04f95-kube-api-access-k9gx4" (OuterVolumeSpecName: "kube-api-access-k9gx4") pod "5930c8e5-111a-4cab-bb13-164ba8b04f95" (UID: "5930c8e5-111a-4cab-bb13-164ba8b04f95"). InnerVolumeSpecName "kube-api-access-k9gx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:42:32.663553 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.663536 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx" Sep 16 07:42:32.669425 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.669409 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:32.749783 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.749756 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9gx4\" (UniqueName: \"kubernetes.io/projected/5930c8e5-111a-4cab-bb13-164ba8b04f95-kube-api-access-k9gx4\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:42:32.749892 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.749784 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5930c8e5-111a-4cab-bb13-164ba8b04f95-extensions-socket-volume\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:42:32.749892 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.749800 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgsc9\" (UniqueName: \"kubernetes.io/projected/7ee4812a-3e41-47eb-8c98-7c7f5ab175ac-kube-api-access-lgsc9\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:42:32.945496 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.945472 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx"] Sep 16 07:42:32.946845 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:42:32.946810 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7a369c_acbf_4bbd_96b8_3a5e3df15c58.slice/crio-1c32e32f895ac6dfac2640da21bbf490eafca4fad278242a00390341672c56cd WatchSource:0}: Error finding container 1c32e32f895ac6dfac2640da21bbf490eafca4fad278242a00390341672c56cd: Status 404 returned error can't find the container with id 1c32e32f895ac6dfac2640da21bbf490eafca4fad278242a00390341672c56cd Sep 16 07:42:32.961252 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:32.961224 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8"] Sep 16 07:42:32.961394 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:42:32.961374 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7914ba72_02eb_4977_be9c_128174d504ba.slice/crio-9d32b88003d348470ac32d2bb1a5d54b05e7854fd67339c022fffcd0bf7dc3ce WatchSource:0}: Error finding container 9d32b88003d348470ac32d2bb1a5d54b05e7854fd67339c022fffcd0bf7dc3ce: Status 404 returned error can't find the container with id 9d32b88003d348470ac32d2bb1a5d54b05e7854fd67339c022fffcd0bf7dc3ce Sep 16 07:42:33.148392 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.148355 2580 generic.go:356] "Generic (PLEG): container finished" podID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" containerID="915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9" exitCode=0 Sep 16 07:42:33.148557 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.148426 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:33.148557 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.148472 2580 scope.go:117] "RemoveContainer" containerID="915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9" Sep 16 07:42:33.150331 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.150304 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" event={"ID":"7914ba72-02eb-4977-be9c-128174d504ba","Type":"ContainerStarted","Data":"f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806"} Sep 16 07:42:33.150458 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.150360 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" event={"ID":"7914ba72-02eb-4977-be9c-128174d504ba","Type":"ContainerStarted","Data":"9d32b88003d348470ac32d2bb1a5d54b05e7854fd67339c022fffcd0bf7dc3ce"} Sep 16 07:42:33.150458 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.150391 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:33.151776 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.151755 2580 generic.go:356] "Generic (PLEG): container finished" podID="5930c8e5-111a-4cab-bb13-164ba8b04f95" containerID="5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60" exitCode=0 Sep 16 07:42:33.151895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.151813 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:33.156874 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:33.156842 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6m5zs\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:33.157634 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.157605 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx" event={"ID":"cb7a369c-acbf-4bbd-96b8-3a5e3df15c58","Type":"ContainerStarted","Data":"8c52f3227c706f3f149376449089693508bf2c42a8f2ed0d99ce7a5c6e4e8352"} Sep 16 07:42:33.157746 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.157656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx" event={"ID":"cb7a369c-acbf-4bbd-96b8-3a5e3df15c58","Type":"ContainerStarted","Data":"1c32e32f895ac6dfac2640da21bbf490eafca4fad278242a00390341672c56cd"} Sep 16 07:42:33.157807 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.157778 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx" Sep 16 07:42:33.159578 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:33.159553 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:33.161806 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.161786 2580 scope.go:117] "RemoveContainer" containerID="915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9" Sep 16 07:42:33.162043 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:33.162025 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9\": container with ID starting with 915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9 not found: ID does not exist" containerID="915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9" Sep 16 07:42:33.162099 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.162049 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9"} err="failed to get container status \"915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9\": rpc error: code = NotFound desc = could not find container \"915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9\": container with ID starting with 915a9d476a64f64c11c39071ce29076a6c64bdb841e425c8ac26fd5c0423d0d9 not found: ID does not exist" Sep 16 07:42:33.162099 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.162065 2580 scope.go:117] "RemoveContainer" containerID="5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60" Sep 16 07:42:33.162733 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:33.162679 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:33.170523 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.170508 2580 scope.go:117] "RemoveContainer" containerID="5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60" Sep 16 07:42:33.170768 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:33.170743 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60\": container with ID starting with 5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60 not found: ID does not exist" containerID="5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60" Sep 16 07:42:33.170840 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.170770 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60"} err="failed to get container status \"5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60\": rpc error: code = NotFound desc = could not find container \"5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60\": container with ID starting with 5a91360d308709bc25c678521884ca8d4d4bb2e64d97317729d2a5b6de399d60 not found: ID does not exist" Sep 16 07:42:33.196574 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.196529 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" podStartSLOduration=1.196511844 podStartE2EDuration="1.196511844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:42:33.19549289 +0000 UTC m=+526.115108836" watchObservedRunningTime="2026-09-16 07:42:33.196511844 +0000 UTC m=+526.116127780" Sep 16 07:42:33.201800 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:33.201773 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6m5zs\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:33.246643 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.246563 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx" podStartSLOduration=1.246549285 podStartE2EDuration="1.246549285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:42:33.246050431 +0000 UTC m=+526.165666377" watchObservedRunningTime="2026-09-16 07:42:33.246549285 +0000 UTC m=+526.166165230" Sep 16 07:42:33.251068 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:33.251044 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6m5zs\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6m5zs" Sep 16 07:42:33.253956 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:33.253930 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rgpxq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rgpxq" Sep 16 07:42:33.560084 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.560009 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5930c8e5-111a-4cab-bb13-164ba8b04f95" path="/var/lib/kubelet/pods/5930c8e5-111a-4cab-bb13-164ba8b04f95/volumes" Sep 16 07:42:33.560363 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:33.560347 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee4812a-3e41-47eb-8c98-7c7f5ab175ac" path="/var/lib/kubelet/pods/7ee4812a-3e41-47eb-8c98-7c7f5ab175ac/volumes" Sep 16 07:42:44.168736 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:44.168661 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:44.168736 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:44.168731 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8q5lx" Sep 16 07:42:48.201893 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:48.201857 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8"] Sep 16 07:42:48.202418 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:48.202097 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" podUID="7914ba72-02eb-4977-be9c-128174d504ba" containerName="manager" containerID="cri-o://f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806" gracePeriod=10 Sep 16 07:42:48.446677 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:48.446652 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:48.469243 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:48.469022 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm6dm\" (UniqueName: \"kubernetes.io/projected/7914ba72-02eb-4977-be9c-128174d504ba-kube-api-access-gm6dm\") pod \"7914ba72-02eb-4977-be9c-128174d504ba\" (UID: \"7914ba72-02eb-4977-be9c-128174d504ba\") " Sep 16 07:42:48.469243 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:48.469101 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7914ba72-02eb-4977-be9c-128174d504ba-extensions-socket-volume\") pod \"7914ba72-02eb-4977-be9c-128174d504ba\" (UID: \"7914ba72-02eb-4977-be9c-128174d504ba\") " Sep 16 07:42:48.469698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:48.469667 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7914ba72-02eb-4977-be9c-128174d504ba-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7914ba72-02eb-4977-be9c-128174d504ba" (UID: "7914ba72-02eb-4977-be9c-128174d504ba"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:42:48.471372 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:48.471341 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7914ba72-02eb-4977-be9c-128174d504ba-kube-api-access-gm6dm" (OuterVolumeSpecName: "kube-api-access-gm6dm") pod "7914ba72-02eb-4977-be9c-128174d504ba" (UID: "7914ba72-02eb-4977-be9c-128174d504ba"). InnerVolumeSpecName "kube-api-access-gm6dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:42:48.570367 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:48.570340 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6dm\" (UniqueName: \"kubernetes.io/projected/7914ba72-02eb-4977-be9c-128174d504ba-kube-api-access-gm6dm\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:42:48.570367 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:48.570361 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7914ba72-02eb-4977-be9c-128174d504ba-extensions-socket-volume\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:42:49.247728 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:49.247693 2580 generic.go:356] "Generic (PLEG): container finished" podID="7914ba72-02eb-4977-be9c-128174d504ba" containerID="f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806" exitCode=0 Sep 16 07:42:49.248131 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:49.247739 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" Sep 16 07:42:49.248131 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:49.247784 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" event={"ID":"7914ba72-02eb-4977-be9c-128174d504ba","Type":"ContainerDied","Data":"f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806"} Sep 16 07:42:49.248131 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:49.247837 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8" event={"ID":"7914ba72-02eb-4977-be9c-128174d504ba","Type":"ContainerDied","Data":"9d32b88003d348470ac32d2bb1a5d54b05e7854fd67339c022fffcd0bf7dc3ce"} Sep 16 07:42:49.248131 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:49.247860 2580 scope.go:117] "RemoveContainer" containerID="f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806" Sep 16 07:42:49.257015 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:49.256998 2580 scope.go:117] "RemoveContainer" containerID="f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806" Sep 16 07:42:49.257320 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:42:49.257301 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806\": container with ID starting with f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806 not found: ID does not exist" containerID="f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806" Sep 16 07:42:49.257392 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:49.257327 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806"} err="failed to get container status \"f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806\": rpc error: code = NotFound desc = could not find container \"f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806\": container with ID starting with f94e4cf12adf41d8ad12324f762a6a26ca6e2b70acb179a7632ce246437d4806 not found: ID does not exist" Sep 16 07:42:49.274825 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:49.274798 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8"] Sep 16 07:42:49.278783 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:49.278762 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4pvc8"] Sep 16 07:42:49.560381 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:42:49.560303 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7914ba72-02eb-4977-be9c-128174d504ba" path="/var/lib/kubelet/pods/7914ba72-02eb-4977-be9c-128174d504ba/volumes" Sep 16 07:43:06.965984 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:06.965950 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lrnjp"] Sep 16 07:43:06.966377 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:06.966362 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7914ba72-02eb-4977-be9c-128174d504ba" containerName="manager" Sep 16 07:43:06.966422 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:06.966378 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7914ba72-02eb-4977-be9c-128174d504ba" containerName="manager" Sep 16 07:43:06.966462 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:06.966449 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7914ba72-02eb-4977-be9c-128174d504ba" containerName="manager" Sep 16 07:43:06.969416 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:06.969400 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:06.971900 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:06.971874 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rz7z2\"" Sep 16 07:43:06.972032 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:06.971874 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 16 07:43:06.976879 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:06.976554 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lrnjp"] Sep 16 07:43:07.006711 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.006687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc06e07e-1569-4e57-82e2-8a3a7a32db37-config-file\") pod \"limitador-limitador-8588749977-lrnjp\" (UID: \"fc06e07e-1569-4e57-82e2-8a3a7a32db37\") " pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:07.006814 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.006741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnrg7\" (UniqueName: \"kubernetes.io/projected/fc06e07e-1569-4e57-82e2-8a3a7a32db37-kube-api-access-wnrg7\") pod \"limitador-limitador-8588749977-lrnjp\" (UID: \"fc06e07e-1569-4e57-82e2-8a3a7a32db37\") " pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:07.073124 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.073093 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lrnjp"] Sep 16 07:43:07.108020 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.107988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrg7\" (UniqueName: \"kubernetes.io/projected/fc06e07e-1569-4e57-82e2-8a3a7a32db37-kube-api-access-wnrg7\") pod \"limitador-limitador-8588749977-lrnjp\" (UID: \"fc06e07e-1569-4e57-82e2-8a3a7a32db37\") " pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:07.108140 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.108067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc06e07e-1569-4e57-82e2-8a3a7a32db37-config-file\") pod \"limitador-limitador-8588749977-lrnjp\" (UID: \"fc06e07e-1569-4e57-82e2-8a3a7a32db37\") " pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:07.108698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.108679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc06e07e-1569-4e57-82e2-8a3a7a32db37-config-file\") pod \"limitador-limitador-8588749977-lrnjp\" (UID: \"fc06e07e-1569-4e57-82e2-8a3a7a32db37\") " pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:07.124484 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.124462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrg7\" (UniqueName: \"kubernetes.io/projected/fc06e07e-1569-4e57-82e2-8a3a7a32db37-kube-api-access-wnrg7\") pod \"limitador-limitador-8588749977-lrnjp\" (UID: \"fc06e07e-1569-4e57-82e2-8a3a7a32db37\") " pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:07.169489 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.169460 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:43:07.171811 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.171795 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" Sep 16 07:43:07.181191 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.181170 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:43:07.208467 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.208442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61ca0c67-45eb-4fe9-965c-eee043f7c7fe-config-file\") pod \"limitador-limitador-84994f7ffc-6tgn9\" (UID: \"61ca0c67-45eb-4fe9-965c-eee043f7c7fe\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" Sep 16 07:43:07.208557 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.208479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtr8\" (UniqueName: \"kubernetes.io/projected/61ca0c67-45eb-4fe9-965c-eee043f7c7fe-kube-api-access-5wtr8\") pod \"limitador-limitador-84994f7ffc-6tgn9\" (UID: \"61ca0c67-45eb-4fe9-965c-eee043f7c7fe\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" Sep 16 07:43:07.214316 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.214296 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:43:07.281288 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.281234 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:07.309133 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.309105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtr8\" (UniqueName: \"kubernetes.io/projected/61ca0c67-45eb-4fe9-965c-eee043f7c7fe-kube-api-access-5wtr8\") pod \"limitador-limitador-84994f7ffc-6tgn9\" (UID: \"61ca0c67-45eb-4fe9-965c-eee043f7c7fe\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" Sep 16 07:43:07.309246 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.309226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61ca0c67-45eb-4fe9-965c-eee043f7c7fe-config-file\") pod \"limitador-limitador-84994f7ffc-6tgn9\" (UID: \"61ca0c67-45eb-4fe9-965c-eee043f7c7fe\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" Sep 16 07:43:07.309749 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.309732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/61ca0c67-45eb-4fe9-965c-eee043f7c7fe-config-file\") pod \"limitador-limitador-84994f7ffc-6tgn9\" (UID: \"61ca0c67-45eb-4fe9-965c-eee043f7c7fe\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" Sep 16 07:43:07.317387 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.317365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtr8\" (UniqueName: \"kubernetes.io/projected/61ca0c67-45eb-4fe9-965c-eee043f7c7fe-kube-api-access-5wtr8\") pod \"limitador-limitador-84994f7ffc-6tgn9\" (UID: \"61ca0c67-45eb-4fe9-965c-eee043f7c7fe\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" Sep 16 07:43:07.482390 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.482357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" Sep 16 07:43:07.550848 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.550818 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lrnjp"] Sep 16 07:43:07.553269 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:43:07.553239 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc06e07e_1569_4e57_82e2_8a3a7a32db37.slice/crio-cee93eb198fa1cdee4b75fc6305f23b20bb9b51af60667ea2eb223d1cff90814 WatchSource:0}: Error finding container cee93eb198fa1cdee4b75fc6305f23b20bb9b51af60667ea2eb223d1cff90814: Status 404 returned error can't find the container with id cee93eb198fa1cdee4b75fc6305f23b20bb9b51af60667ea2eb223d1cff90814 Sep 16 07:43:07.751068 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.751035 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:43:07.752538 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:43:07.752513 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ca0c67_45eb_4fe9_965c_eee043f7c7fe.slice/crio-3e649ab134111b1101eb3cedd434c26b37b33e1b03ea503a00aa42541b4ecbd7 WatchSource:0}: Error finding container 3e649ab134111b1101eb3cedd434c26b37b33e1b03ea503a00aa42541b4ecbd7: Status 404 returned error can't find the container with id 3e649ab134111b1101eb3cedd434c26b37b33e1b03ea503a00aa42541b4ecbd7 Sep 16 07:43:07.797887 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.797824 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n292t"] Sep 16 07:43:07.800940 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.800915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" Sep 16 07:43:07.803520 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.803498 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j7cgc\"" Sep 16 07:43:07.809646 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.809624 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n292t"] Sep 16 07:43:07.913642 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:07.913618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlf82\" (UniqueName: \"kubernetes.io/projected/2233efbe-8ea8-4160-81cc-78d333ba43f0-kube-api-access-jlf82\") pod \"authorino-7bcfd8f9cd-n292t\" (UID: \"2233efbe-8ea8-4160-81cc-78d333ba43f0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" Sep 16 07:43:08.014955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:08.014927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlf82\" (UniqueName: \"kubernetes.io/projected/2233efbe-8ea8-4160-81cc-78d333ba43f0-kube-api-access-jlf82\") pod \"authorino-7bcfd8f9cd-n292t\" (UID: \"2233efbe-8ea8-4160-81cc-78d333ba43f0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" Sep 16 07:43:08.023397 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:08.023377 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlf82\" (UniqueName: \"kubernetes.io/projected/2233efbe-8ea8-4160-81cc-78d333ba43f0-kube-api-access-jlf82\") pod \"authorino-7bcfd8f9cd-n292t\" (UID: \"2233efbe-8ea8-4160-81cc-78d333ba43f0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" Sep 16 07:43:08.111017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:08.110963 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" Sep 16 07:43:08.362718 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:08.362637 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" event={"ID":"fc06e07e-1569-4e57-82e2-8a3a7a32db37","Type":"ContainerStarted","Data":"cee93eb198fa1cdee4b75fc6305f23b20bb9b51af60667ea2eb223d1cff90814"} Sep 16 07:43:08.363919 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:08.363890 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" event={"ID":"61ca0c67-45eb-4fe9-965c-eee043f7c7fe","Type":"ContainerStarted","Data":"3e649ab134111b1101eb3cedd434c26b37b33e1b03ea503a00aa42541b4ecbd7"} Sep 16 07:43:08.377382 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:08.377362 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n292t"] Sep 16 07:43:08.379117 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:43:08.379094 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2233efbe_8ea8_4160_81cc_78d333ba43f0.slice/crio-d62a6bbf3347d45fc59ef8678e331d5af321a1de36ec0fc6afc26c3af5051244 WatchSource:0}: Error finding container d62a6bbf3347d45fc59ef8678e331d5af321a1de36ec0fc6afc26c3af5051244: Status 404 returned error can't find the container with id d62a6bbf3347d45fc59ef8678e331d5af321a1de36ec0fc6afc26c3af5051244 Sep 16 07:43:09.376117 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:09.376078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" event={"ID":"2233efbe-8ea8-4160-81cc-78d333ba43f0","Type":"ContainerStarted","Data":"d62a6bbf3347d45fc59ef8678e331d5af321a1de36ec0fc6afc26c3af5051244"} Sep 16 07:43:09.377720 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:09.377694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" event={"ID":"fc06e07e-1569-4e57-82e2-8a3a7a32db37","Type":"ContainerStarted","Data":"f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b"} Sep 16 07:43:09.377836 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:09.377800 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:09.379308 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:09.379288 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" event={"ID":"61ca0c67-45eb-4fe9-965c-eee043f7c7fe","Type":"ContainerStarted","Data":"2370f4cfc63133327318255f89cd5d5afe8ea1286a3becba1dcb859970120a29"} Sep 16 07:43:09.379406 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:09.379372 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" Sep 16 07:43:09.395224 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:09.395166 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" podStartSLOduration=1.787181462 podStartE2EDuration="3.395154028s" totalImagesPullingTime="1.607972566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:43:09.393033307 +0000 UTC m=+562.312649254" watchObservedRunningTime="2026-09-16 07:43:09.395154028 +0000 UTC m=+562.314769974" Sep 16 07:43:09.408692 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:09.408655 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" podStartSLOduration=1.194287613 podStartE2EDuration="2.408643041s" totalImagesPullingTime="1.214355428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:43:09.407325855 +0000 UTC m=+562.326941800" watchObservedRunningTime="2026-09-16 07:43:09.408643041 +0000 UTC m=+562.328258988" Sep 16 07:43:11.330397 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:11.330355 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n292t"] Sep 16 07:43:13.404294 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:13.404258 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" event={"ID":"2233efbe-8ea8-4160-81cc-78d333ba43f0","Type":"ContainerStarted","Data":"3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c"} Sep 16 07:43:13.404294 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:13.404284 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" podUID="2233efbe-8ea8-4160-81cc-78d333ba43f0" containerName="authorino" containerID="cri-o://3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c" gracePeriod=30 Sep 16 07:43:13.418727 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:13.418681 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" podStartSLOduration=2.409037074 podStartE2EDuration="6.418667078s" totalImagesPullingTime="4.009630004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:43:13.418026787 +0000 UTC m=+566.337642734" watchObservedRunningTime="2026-09-16 07:43:13.418667078 +0000 UTC m=+566.338283090" Sep 16 07:43:13.648015 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:13.647992 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" Sep 16 07:43:13.760362 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:13.760303 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlf82\" (UniqueName: \"kubernetes.io/projected/2233efbe-8ea8-4160-81cc-78d333ba43f0-kube-api-access-jlf82\") pod \"2233efbe-8ea8-4160-81cc-78d333ba43f0\" (UID: \"2233efbe-8ea8-4160-81cc-78d333ba43f0\") " Sep 16 07:43:13.762239 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:13.762198 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2233efbe-8ea8-4160-81cc-78d333ba43f0-kube-api-access-jlf82" (OuterVolumeSpecName: "kube-api-access-jlf82") pod "2233efbe-8ea8-4160-81cc-78d333ba43f0" (UID: "2233efbe-8ea8-4160-81cc-78d333ba43f0"). InnerVolumeSpecName "kube-api-access-jlf82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:43:13.861061 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:13.861039 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlf82\" (UniqueName: \"kubernetes.io/projected/2233efbe-8ea8-4160-81cc-78d333ba43f0-kube-api-access-jlf82\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:43:14.410955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:14.410915 2580 generic.go:356] "Generic (PLEG): container finished" podID="2233efbe-8ea8-4160-81cc-78d333ba43f0" containerID="3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c" exitCode=0 Sep 16 07:43:14.411358 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:14.410960 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" Sep 16 07:43:14.411358 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:14.410965 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" event={"ID":"2233efbe-8ea8-4160-81cc-78d333ba43f0","Type":"ContainerDied","Data":"3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c"} Sep 16 07:43:14.411358 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:14.410998 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n292t" event={"ID":"2233efbe-8ea8-4160-81cc-78d333ba43f0","Type":"ContainerDied","Data":"d62a6bbf3347d45fc59ef8678e331d5af321a1de36ec0fc6afc26c3af5051244"} Sep 16 07:43:14.411358 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:14.411016 2580 scope.go:117] "RemoveContainer" containerID="3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c" Sep 16 07:43:14.420198 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:14.420179 2580 scope.go:117] "RemoveContainer" containerID="3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c" Sep 16 07:43:14.420487 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:43:14.420469 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c\": container with ID starting with 3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c not found: ID does not exist" containerID="3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c" Sep 16 07:43:14.420534 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:14.420495 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c"} err="failed to get container status \"3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c\": rpc error: code = NotFound desc = could not find container \"3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c\": container with ID starting with 3f4ed373780236ba73ed723d923fa4a1c6ad0a3703b98419ccca9d37dd63070c not found: ID does not exist" Sep 16 07:43:14.433644 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:14.433623 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n292t"] Sep 16 07:43:14.437800 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:14.437780 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n292t"] Sep 16 07:43:15.560053 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:15.560019 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2233efbe-8ea8-4160-81cc-78d333ba43f0" path="/var/lib/kubelet/pods/2233efbe-8ea8-4160-81cc-78d333ba43f0/volumes" Sep 16 07:43:20.387029 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:20.386997 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6tgn9" Sep 16 07:43:20.387682 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:20.387083 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:20.452866 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:20.452830 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lrnjp"] Sep 16 07:43:20.453147 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:20.453118 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" podUID="fc06e07e-1569-4e57-82e2-8a3a7a32db37" containerName="limitador" containerID="cri-o://f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b" gracePeriod=30 Sep 16 07:43:21.009883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.009858 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:21.117016 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.116954 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnrg7\" (UniqueName: \"kubernetes.io/projected/fc06e07e-1569-4e57-82e2-8a3a7a32db37-kube-api-access-wnrg7\") pod \"fc06e07e-1569-4e57-82e2-8a3a7a32db37\" (UID: \"fc06e07e-1569-4e57-82e2-8a3a7a32db37\") " Sep 16 07:43:21.117016 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.117003 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc06e07e-1569-4e57-82e2-8a3a7a32db37-config-file\") pod \"fc06e07e-1569-4e57-82e2-8a3a7a32db37\" (UID: \"fc06e07e-1569-4e57-82e2-8a3a7a32db37\") " Sep 16 07:43:21.117328 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.117306 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc06e07e-1569-4e57-82e2-8a3a7a32db37-config-file" (OuterVolumeSpecName: "config-file") pod "fc06e07e-1569-4e57-82e2-8a3a7a32db37" (UID: "fc06e07e-1569-4e57-82e2-8a3a7a32db37"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:43:21.118980 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.118959 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc06e07e-1569-4e57-82e2-8a3a7a32db37-kube-api-access-wnrg7" (OuterVolumeSpecName: "kube-api-access-wnrg7") pod "fc06e07e-1569-4e57-82e2-8a3a7a32db37" (UID: "fc06e07e-1569-4e57-82e2-8a3a7a32db37"). InnerVolumeSpecName "kube-api-access-wnrg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:43:21.217683 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.217658 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnrg7\" (UniqueName: \"kubernetes.io/projected/fc06e07e-1569-4e57-82e2-8a3a7a32db37-kube-api-access-wnrg7\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:43:21.217683 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.217679 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc06e07e-1569-4e57-82e2-8a3a7a32db37-config-file\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:43:21.453084 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.453052 2580 generic.go:356] "Generic (PLEG): container finished" podID="fc06e07e-1569-4e57-82e2-8a3a7a32db37" containerID="f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b" exitCode=0 Sep 16 07:43:21.453531 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.453100 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" Sep 16 07:43:21.453531 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.453117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" event={"ID":"fc06e07e-1569-4e57-82e2-8a3a7a32db37","Type":"ContainerDied","Data":"f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b"} Sep 16 07:43:21.453531 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.453166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lrnjp" event={"ID":"fc06e07e-1569-4e57-82e2-8a3a7a32db37","Type":"ContainerDied","Data":"cee93eb198fa1cdee4b75fc6305f23b20bb9b51af60667ea2eb223d1cff90814"} Sep 16 07:43:21.453531 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.453190 2580 scope.go:117] "RemoveContainer" containerID="f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b" Sep 16 07:43:21.462238 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.462206 2580 scope.go:117] "RemoveContainer" containerID="f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b" Sep 16 07:43:21.462514 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:43:21.462495 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b\": container with ID starting with f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b not found: ID does not exist" containerID="f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b" Sep 16 07:43:21.462580 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.462524 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b"} err="failed to get container status \"f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b\": rpc error: code = NotFound desc = could not find container \"f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b\": container with ID starting with f647099964b5d60cd1f41d512da7738a8f49d1023bd88e0928d9538728b6e69b not found: ID does not exist" Sep 16 07:43:21.474987 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.474965 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lrnjp"] Sep 16 07:43:21.480242 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.480219 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lrnjp"] Sep 16 07:43:21.559542 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:21.559519 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc06e07e-1569-4e57-82e2-8a3a7a32db37" path="/var/lib/kubelet/pods/fc06e07e-1569-4e57-82e2-8a3a7a32db37/volumes" Sep 16 07:43:23.472032 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.471992 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-682z2"] Sep 16 07:43:23.472521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.472388 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2233efbe-8ea8-4160-81cc-78d333ba43f0" containerName="authorino" Sep 16 07:43:23.472521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.472402 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2233efbe-8ea8-4160-81cc-78d333ba43f0" containerName="authorino" Sep 16 07:43:23.472521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.472412 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc06e07e-1569-4e57-82e2-8a3a7a32db37" containerName="limitador" Sep 16 07:43:23.472521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.472417 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc06e07e-1569-4e57-82e2-8a3a7a32db37" containerName="limitador" Sep 16 07:43:23.472521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.472490 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc06e07e-1569-4e57-82e2-8a3a7a32db37" containerName="limitador" Sep 16 07:43:23.472521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.472504 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2233efbe-8ea8-4160-81cc-78d333ba43f0" containerName="authorino" Sep 16 07:43:23.476958 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.476939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" Sep 16 07:43:23.479725 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.479701 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 16 07:43:23.480024 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.480002 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ppl47\"" Sep 16 07:43:23.481383 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.481360 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 07:43:23.481619 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.481598 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 07:43:23.484140 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.484112 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-682z2"] Sep 16 07:43:23.537918 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.537891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75c1db6e-0e1a-4e55-9636-e1c826ffc392-data\") pod \"postgres-69fb4594cf-682z2\" (UID: \"75c1db6e-0e1a-4e55-9636-e1c826ffc392\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" Sep 16 07:43:23.538023 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.537931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65bth\" (UniqueName: \"kubernetes.io/projected/75c1db6e-0e1a-4e55-9636-e1c826ffc392-kube-api-access-65bth\") pod \"postgres-69fb4594cf-682z2\" (UID: \"75c1db6e-0e1a-4e55-9636-e1c826ffc392\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" Sep 16 07:43:23.639087 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.639062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75c1db6e-0e1a-4e55-9636-e1c826ffc392-data\") pod \"postgres-69fb4594cf-682z2\" (UID: \"75c1db6e-0e1a-4e55-9636-e1c826ffc392\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" Sep 16 07:43:23.639224 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.639089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bth\" (UniqueName: \"kubernetes.io/projected/75c1db6e-0e1a-4e55-9636-e1c826ffc392-kube-api-access-65bth\") pod \"postgres-69fb4594cf-682z2\" (UID: \"75c1db6e-0e1a-4e55-9636-e1c826ffc392\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" Sep 16 07:43:23.639511 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.639490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75c1db6e-0e1a-4e55-9636-e1c826ffc392-data\") pod \"postgres-69fb4594cf-682z2\" (UID: \"75c1db6e-0e1a-4e55-9636-e1c826ffc392\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" Sep 16 07:43:23.647226 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.647198 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bth\" (UniqueName: \"kubernetes.io/projected/75c1db6e-0e1a-4e55-9636-e1c826ffc392-kube-api-access-65bth\") pod \"postgres-69fb4594cf-682z2\" (UID: \"75c1db6e-0e1a-4e55-9636-e1c826ffc392\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" Sep 16 07:43:23.789346 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:23.789293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" Sep 16 07:43:24.059750 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:24.059725 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-682z2"] Sep 16 07:43:24.060842 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:43:24.060814 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c1db6e_0e1a_4e55_9636_e1c826ffc392.slice/crio-d89b8a8ad5d3af4db21dd4bae6e2683495c95f7ff6c5d13d17cb56f63975c42c WatchSource:0}: Error finding container d89b8a8ad5d3af4db21dd4bae6e2683495c95f7ff6c5d13d17cb56f63975c42c: Status 404 returned error can't find the container with id d89b8a8ad5d3af4db21dd4bae6e2683495c95f7ff6c5d13d17cb56f63975c42c Sep 16 07:43:24.471940 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:24.471901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" event={"ID":"75c1db6e-0e1a-4e55-9636-e1c826ffc392","Type":"ContainerStarted","Data":"d89b8a8ad5d3af4db21dd4bae6e2683495c95f7ff6c5d13d17cb56f63975c42c"} Sep 16 07:43:29.505844 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:29.505804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" event={"ID":"75c1db6e-0e1a-4e55-9636-e1c826ffc392","Type":"ContainerStarted","Data":"ff0f75c2a8e3fef066a182dba79fde925a707948750dcf01bdfeba30eae4a9bd"} Sep 16 07:43:29.506337 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:29.505870 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" Sep 16 07:43:29.521785 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:29.521713 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" podStartSLOduration=1.8363349 podStartE2EDuration="6.521697288s" totalImagesPullingTime="4.685362388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:43:29.520632207 +0000 UTC m=+582.440248154" watchObservedRunningTime="2026-09-16 07:43:29.521697288 +0000 UTC m=+582.441313234" Sep 16 07:43:35.540075 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:35.540044 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-682z2" Sep 16 07:43:36.069897 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.069856 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-nqc6m"] Sep 16 07:43:36.075999 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.075980 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nqc6m" Sep 16 07:43:36.078491 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.078467 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j7cgc\"" Sep 16 07:43:36.079656 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.079633 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-nqc6m"] Sep 16 07:43:36.129008 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.128983 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nqc6m"] Sep 16 07:43:36.129235 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:43:36.129204 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-r69hn], unattached volumes=[], failed to process volumes=[kube-api-access-r69hn]: context canceled" pod="kuadrant-system/authorino-557f876f69-nqc6m" podUID="8b626c18-10bf-41e7-9d41-433778aa0359" Sep 16 07:43:36.151063 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.151039 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-c26w7"] Sep 16 07:43:36.155146 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.155131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-c26w7" Sep 16 07:43:36.157558 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.157542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 16 07:43:36.160837 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.160817 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-c26w7"] Sep 16 07:43:36.244369 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.244344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69hn\" (UniqueName: \"kubernetes.io/projected/8b626c18-10bf-41e7-9d41-433778aa0359-kube-api-access-r69hn\") pod \"authorino-557f876f69-nqc6m\" (UID: \"8b626c18-10bf-41e7-9d41-433778aa0359\") " pod="kuadrant-system/authorino-557f876f69-nqc6m" Sep 16 07:43:36.244481 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.244394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhz7v\" (UniqueName: \"kubernetes.io/projected/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-kube-api-access-hhz7v\") pod \"authorino-6d4dfc649f-c26w7\" (UID: \"2bcddf01-eaca-4b1f-83c9-e065ae7c7051\") " pod="kuadrant-system/authorino-6d4dfc649f-c26w7" Sep 16 07:43:36.244481 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.244417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-tls-cert\") pod \"authorino-6d4dfc649f-c26w7\" (UID: \"2bcddf01-eaca-4b1f-83c9-e065ae7c7051\") " pod="kuadrant-system/authorino-6d4dfc649f-c26w7" Sep 16 07:43:36.292891 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.292869 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-c26w7"] Sep 16 07:43:36.293077 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:43:36.293060 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hhz7v tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-c26w7" podUID="2bcddf01-eaca-4b1f-83c9-e065ae7c7051" Sep 16 07:43:36.345782 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.345723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhz7v\" (UniqueName: \"kubernetes.io/projected/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-kube-api-access-hhz7v\") pod \"authorino-6d4dfc649f-c26w7\" (UID: \"2bcddf01-eaca-4b1f-83c9-e065ae7c7051\") " pod="kuadrant-system/authorino-6d4dfc649f-c26w7" Sep 16 07:43:36.345782 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.345754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-tls-cert\") pod \"authorino-6d4dfc649f-c26w7\" (UID: \"2bcddf01-eaca-4b1f-83c9-e065ae7c7051\") " pod="kuadrant-system/authorino-6d4dfc649f-c26w7" Sep 16 07:43:36.345905 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.345795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69hn\" (UniqueName: \"kubernetes.io/projected/8b626c18-10bf-41e7-9d41-433778aa0359-kube-api-access-r69hn\") pod \"authorino-557f876f69-nqc6m\" (UID: \"8b626c18-10bf-41e7-9d41-433778aa0359\") " pod="kuadrant-system/authorino-557f876f69-nqc6m" Sep 16 07:43:36.348226 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.348188 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-tls-cert\") pod \"authorino-6d4dfc649f-c26w7\" (UID: \"2bcddf01-eaca-4b1f-83c9-e065ae7c7051\") " pod="kuadrant-system/authorino-6d4dfc649f-c26w7" Sep 16 07:43:36.353333 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.353307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69hn\" (UniqueName: \"kubernetes.io/projected/8b626c18-10bf-41e7-9d41-433778aa0359-kube-api-access-r69hn\") pod \"authorino-557f876f69-nqc6m\" (UID: \"8b626c18-10bf-41e7-9d41-433778aa0359\") " pod="kuadrant-system/authorino-557f876f69-nqc6m" Sep 16 07:43:36.353621 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.353600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhz7v\" (UniqueName: \"kubernetes.io/projected/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-kube-api-access-hhz7v\") pod \"authorino-6d4dfc649f-c26w7\" (UID: \"2bcddf01-eaca-4b1f-83c9-e065ae7c7051\") " pod="kuadrant-system/authorino-6d4dfc649f-c26w7" Sep 16 07:43:36.544407 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.544381 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-c26w7" Sep 16 07:43:36.544769 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.544385 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nqc6m" Sep 16 07:43:36.552336 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.552317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-c26w7" Sep 16 07:43:36.555039 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.555024 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nqc6m" Sep 16 07:43:36.648660 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.648588 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhz7v\" (UniqueName: \"kubernetes.io/projected/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-kube-api-access-hhz7v\") pod \"2bcddf01-eaca-4b1f-83c9-e065ae7c7051\" (UID: \"2bcddf01-eaca-4b1f-83c9-e065ae7c7051\") " Sep 16 07:43:36.648660 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.648647 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r69hn\" (UniqueName: \"kubernetes.io/projected/8b626c18-10bf-41e7-9d41-433778aa0359-kube-api-access-r69hn\") pod \"8b626c18-10bf-41e7-9d41-433778aa0359\" (UID: \"8b626c18-10bf-41e7-9d41-433778aa0359\") " Sep 16 07:43:36.648849 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.648675 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-tls-cert\") pod \"2bcddf01-eaca-4b1f-83c9-e065ae7c7051\" (UID: \"2bcddf01-eaca-4b1f-83c9-e065ae7c7051\") " Sep 16 07:43:36.650771 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.650743 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2bcddf01-eaca-4b1f-83c9-e065ae7c7051" (UID: "2bcddf01-eaca-4b1f-83c9-e065ae7c7051"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:43:36.650890 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.650789 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b626c18-10bf-41e7-9d41-433778aa0359-kube-api-access-r69hn" (OuterVolumeSpecName: "kube-api-access-r69hn") pod "8b626c18-10bf-41e7-9d41-433778aa0359" (UID: "8b626c18-10bf-41e7-9d41-433778aa0359"). InnerVolumeSpecName "kube-api-access-r69hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:43:36.650890 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.650836 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-kube-api-access-hhz7v" (OuterVolumeSpecName: "kube-api-access-hhz7v") pod "2bcddf01-eaca-4b1f-83c9-e065ae7c7051" (UID: "2bcddf01-eaca-4b1f-83c9-e065ae7c7051"). InnerVolumeSpecName "kube-api-access-hhz7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:43:36.749908 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.749868 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-tls-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:43:36.749908 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.749904 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhz7v\" (UniqueName: \"kubernetes.io/projected/2bcddf01-eaca-4b1f-83c9-e065ae7c7051-kube-api-access-hhz7v\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:43:36.750062 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:36.749919 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r69hn\" (UniqueName: \"kubernetes.io/projected/8b626c18-10bf-41e7-9d41-433778aa0359-kube-api-access-r69hn\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:43:37.550116 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:37.550083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-c26w7" Sep 16 07:43:37.550546 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:37.550083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nqc6m" Sep 16 07:43:37.584440 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:37.584413 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nqc6m"] Sep 16 07:43:37.589853 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:37.589825 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-nqc6m"] Sep 16 07:43:37.611101 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:37.611074 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-c26w7"] Sep 16 07:43:37.614750 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:37.614724 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-c26w7"] Sep 16 07:43:39.560598 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:39.560560 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bcddf01-eaca-4b1f-83c9-e065ae7c7051" path="/var/lib/kubelet/pods/2bcddf01-eaca-4b1f-83c9-e065ae7c7051/volumes" Sep 16 07:43:39.560949 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:39.560786 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b626c18-10bf-41e7-9d41-433778aa0359" path="/var/lib/kubelet/pods/8b626c18-10bf-41e7-9d41-433778aa0359/volumes" Sep 16 07:43:43.194107 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.194069 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-5tt6v"] Sep 16 07:43:43.196598 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.196582 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.200550 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.200521 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 07:43:43.200693 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.200582 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 07:43:43.200693 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.200687 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 07:43:43.200804 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.200755 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nvkmk\"" Sep 16 07:43:43.209549 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.209529 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-5tt6v"] Sep 16 07:43:43.300040 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.300009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-metrics-certs\") pod \"maas-controller-bbd9d7d98-5tt6v\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.300158 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.300061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvms7\" (UniqueName: \"kubernetes.io/projected/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-kube-api-access-qvms7\") pod \"maas-controller-bbd9d7d98-5tt6v\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.300158 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.300106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-webhook-cert\") pod \"maas-controller-bbd9d7d98-5tt6v\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.401310 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.401280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvms7\" (UniqueName: \"kubernetes.io/projected/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-kube-api-access-qvms7\") pod \"maas-controller-bbd9d7d98-5tt6v\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.401404 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.401319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-webhook-cert\") pod \"maas-controller-bbd9d7d98-5tt6v\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.401404 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.401384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-metrics-certs\") pod \"maas-controller-bbd9d7d98-5tt6v\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.403857 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.403833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-metrics-certs\") pod \"maas-controller-bbd9d7d98-5tt6v\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.403949 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.403833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-webhook-cert\") pod \"maas-controller-bbd9d7d98-5tt6v\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.409157 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.409133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvms7\" (UniqueName: \"kubernetes.io/projected/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-kube-api-access-qvms7\") pod \"maas-controller-bbd9d7d98-5tt6v\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.506867 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.506807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:43.776050 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:43.776022 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-5tt6v"] Sep 16 07:43:43.777545 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:43:43.777520 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cccfa2_901c_4ff8_a8c5_f4af1ad249a9.slice/crio-1f02c3aef56c44da3a7ba2667ef9e8532c341e1fb64d8ff3dfd8b56e4d0639f4 WatchSource:0}: Error finding container 1f02c3aef56c44da3a7ba2667ef9e8532c341e1fb64d8ff3dfd8b56e4d0639f4: Status 404 returned error can't find the container with id 1f02c3aef56c44da3a7ba2667ef9e8532c341e1fb64d8ff3dfd8b56e4d0639f4 Sep 16 07:43:44.592571 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:44.592529 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" event={"ID":"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9","Type":"ContainerStarted","Data":"1f02c3aef56c44da3a7ba2667ef9e8532c341e1fb64d8ff3dfd8b56e4d0639f4"} Sep 16 07:43:46.610956 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:46.610915 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" event={"ID":"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9","Type":"ContainerStarted","Data":"6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753"} Sep 16 07:43:46.611564 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:46.611533 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:43:46.631667 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:46.631615 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" podStartSLOduration=1.411802943 podStartE2EDuration="3.631602707s" totalImagesPullingTime="2.219799764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:43:46.629238212 +0000 UTC m=+599.548854158" watchObservedRunningTime="2026-09-16 07:43:46.631602707 +0000 UTC m=+599.551218653" Sep 16 07:43:47.463793 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:47.463763 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:43:47.463964 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:47.463769 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:43:57.618360 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:43:57.618329 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 07:44:19.400797 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.400724 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9"] Sep 16 07:44:19.413533 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.413506 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:19.414242 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.414202 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9"] Sep 16 07:44:19.416078 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.416055 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 16 07:44:19.416202 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.416066 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fknxb\"" Sep 16 07:44:19.416485 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.416461 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 16 07:44:19.494871 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.494835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-metrics-certs\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:19.495000 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.494880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/649a9983-4492-428b-9e73-bc09c7042fbb-openshift-service-ca\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:19.495000 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.494924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrk68\" (UniqueName: \"kubernetes.io/projected/649a9983-4492-428b-9e73-bc09c7042fbb-kube-api-access-xrk68\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:19.495092 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.495016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-maas-api-tls\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:19.596065 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.596035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-metrics-certs\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:19.596195 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.596072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/649a9983-4492-428b-9e73-bc09c7042fbb-openshift-service-ca\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:19.596195 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.596096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrk68\" (UniqueName: \"kubernetes.io/projected/649a9983-4492-428b-9e73-bc09c7042fbb-kube-api-access-xrk68\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:19.596195 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.596136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-maas-api-tls\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:19.596343 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:44:19.596191 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 16 07:44:19.596343 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:44:19.596271 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Sep 16 07:44:19.596343 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:44:19.596313 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-metrics-certs podName:649a9983-4492-428b-9e73-bc09c7042fbb nodeName:}" failed. No retries permitted until 2026-09-16 07:44:20.096291472 +0000 UTC m=+633.015907402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-metrics-certs") pod "maas-api-66d85f5b4d-55gz9" (UID: "649a9983-4492-428b-9e73-bc09c7042fbb") : secret "maas-api-metrics-tls" not found Sep 16 07:44:19.596343 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:44:19.596334 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-maas-api-tls podName:649a9983-4492-428b-9e73-bc09c7042fbb nodeName:}" failed. No retries permitted until 2026-09-16 07:44:20.096325232 +0000 UTC m=+633.015941166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-maas-api-tls") pod "maas-api-66d85f5b4d-55gz9" (UID: "649a9983-4492-428b-9e73-bc09c7042fbb") : secret "maas-api-serving-cert" not found Sep 16 07:44:19.596777 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.596758 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/649a9983-4492-428b-9e73-bc09c7042fbb-openshift-service-ca\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:19.604720 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:19.604693 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrk68\" (UniqueName: \"kubernetes.io/projected/649a9983-4492-428b-9e73-bc09c7042fbb-kube-api-access-xrk68\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:20.022815 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.022782 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:44:20.027774 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.027747 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fh9w6"] Sep 16 07:44:20.031661 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.031640 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" Sep 16 07:44:20.035182 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.035161 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jfvmp\"" Sep 16 07:44:20.035444 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.035430 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 16 07:44:20.046355 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.046333 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fh9w6"] Sep 16 07:44:20.100554 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.100527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945b9e69-a7b4-458a-a638-5a9ffcc8cd8e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fh9w6\" (UID: \"945b9e69-a7b4-458a-a638-5a9ffcc8cd8e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" Sep 16 07:44:20.100695 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.100668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-metrics-certs\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:20.100795 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.100775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-maas-api-tls\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:20.100854 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.100822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkrm\" (UniqueName: \"kubernetes.io/projected/945b9e69-a7b4-458a-a638-5a9ffcc8cd8e-kube-api-access-zgkrm\") pod \"payload-processing-5b54b7fbb7-fh9w6\" (UID: \"945b9e69-a7b4-458a-a638-5a9ffcc8cd8e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" Sep 16 07:44:20.103458 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.103431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-metrics-certs\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:20.103613 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.103591 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-maas-api-tls\") pod \"maas-api-66d85f5b4d-55gz9\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:20.201716 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.201675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkrm\" (UniqueName: \"kubernetes.io/projected/945b9e69-a7b4-458a-a638-5a9ffcc8cd8e-kube-api-access-zgkrm\") pod \"payload-processing-5b54b7fbb7-fh9w6\" (UID: \"945b9e69-a7b4-458a-a638-5a9ffcc8cd8e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" Sep 16 07:44:20.201887 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.201739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945b9e69-a7b4-458a-a638-5a9ffcc8cd8e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fh9w6\" (UID: \"945b9e69-a7b4-458a-a638-5a9ffcc8cd8e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" Sep 16 07:44:20.202472 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.202448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945b9e69-a7b4-458a-a638-5a9ffcc8cd8e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-fh9w6\" (UID: \"945b9e69-a7b4-458a-a638-5a9ffcc8cd8e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" Sep 16 07:44:20.211888 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.211864 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkrm\" (UniqueName: \"kubernetes.io/projected/945b9e69-a7b4-458a-a638-5a9ffcc8cd8e-kube-api-access-zgkrm\") pod \"payload-processing-5b54b7fbb7-fh9w6\" (UID: \"945b9e69-a7b4-458a-a638-5a9ffcc8cd8e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" Sep 16 07:44:20.324819 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.324730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:20.342777 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.342754 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" Sep 16 07:44:20.598035 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.598006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9"] Sep 16 07:44:20.598995 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:44:20.598972 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649a9983_4492_428b_9e73_bc09c7042fbb.slice/crio-bba778f4c5524d468a2cd06740f32c7244efb00bbe0a2381564f983d3017d626 WatchSource:0}: Error finding container bba778f4c5524d468a2cd06740f32c7244efb00bbe0a2381564f983d3017d626: Status 404 returned error can't find the container with id bba778f4c5524d468a2cd06740f32c7244efb00bbe0a2381564f983d3017d626 Sep 16 07:44:20.600372 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.600353 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:44:20.623337 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.623316 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-fh9w6"] Sep 16 07:44:20.624412 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:44:20.624392 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945b9e69_a7b4_458a_a638_5a9ffcc8cd8e.slice/crio-9a3ca145129999212c630e7ccd4bd990d3fe5ba2cea432935d32b32cae96d0aa WatchSource:0}: Error finding container 9a3ca145129999212c630e7ccd4bd990d3fe5ba2cea432935d32b32cae96d0aa: Status 404 returned error can't find the container with id 9a3ca145129999212c630e7ccd4bd990d3fe5ba2cea432935d32b32cae96d0aa Sep 16 07:44:20.803711 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.803671 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" event={"ID":"945b9e69-a7b4-458a-a638-5a9ffcc8cd8e","Type":"ContainerStarted","Data":"9a3ca145129999212c630e7ccd4bd990d3fe5ba2cea432935d32b32cae96d0aa"} Sep 16 07:44:20.804979 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:20.804948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" event={"ID":"649a9983-4492-428b-9e73-bc09c7042fbb","Type":"ContainerStarted","Data":"bba778f4c5524d468a2cd06740f32c7244efb00bbe0a2381564f983d3017d626"} Sep 16 07:44:22.820078 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:22.820021 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" event={"ID":"945b9e69-a7b4-458a-a638-5a9ffcc8cd8e","Type":"ContainerStarted","Data":"b630b86bbd4499870c45807f04c33d1424dfc73d63e1ec37b63162360b7e5d4a"} Sep 16 07:44:22.820613 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:22.820162 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" Sep 16 07:44:22.822010 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:22.821956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" event={"ID":"649a9983-4492-428b-9e73-bc09c7042fbb","Type":"ContainerStarted","Data":"649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155"} Sep 16 07:44:22.822136 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:22.822015 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:22.837967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:22.837927 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" podStartSLOduration=1.104808986 podStartE2EDuration="2.837914031s" totalImagesPullingTime="1.733105045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:44:22.836168046 +0000 UTC m=+635.755783992" watchObservedRunningTime="2026-09-16 07:44:22.837914031 +0000 UTC m=+635.757529977" Sep 16 07:44:22.853255 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:22.853199 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" podStartSLOduration=2.095650072 podStartE2EDuration="3.853184502s" totalImagesPullingTime="1.75753443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:44:22.851346121 +0000 UTC m=+635.770962059" watchObservedRunningTime="2026-09-16 07:44:22.853184502 +0000 UTC m=+635.772800491" Sep 16 07:44:28.836206 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:28.836175 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 07:44:33.833416 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:33.833381 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-fh9w6" Sep 16 07:44:41.697167 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:41.697131 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:44:47.399649 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:47.399614 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:44:52.103317 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:44:52.103281 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:45:00.798809 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:45:00.798773 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:45:16.096791 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:45:16.096757 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:45:33.223861 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:45:33.223825 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:45:46.712743 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:45:46.712664 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:46:30.062950 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.062907 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58458bb776-79rkb"] Sep 16 07:46:30.066509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.066489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58458bb776-79rkb" Sep 16 07:46:30.070691 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.070634 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j7cgc\"" Sep 16 07:46:30.070852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.070808 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 16 07:46:30.074495 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.074433 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58458bb776-79rkb"] Sep 16 07:46:30.138228 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.138181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpgkf\" (UniqueName: \"kubernetes.io/projected/df2f5e81-9bba-4cd7-afdf-956188daa9ed-kube-api-access-fpgkf\") pod \"authorino-58458bb776-79rkb\" (UID: \"df2f5e81-9bba-4cd7-afdf-956188daa9ed\") " pod="kuadrant-system/authorino-58458bb776-79rkb" Sep 16 07:46:30.138352 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.138240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df2f5e81-9bba-4cd7-afdf-956188daa9ed-tls-cert\") pod \"authorino-58458bb776-79rkb\" (UID: \"df2f5e81-9bba-4cd7-afdf-956188daa9ed\") " pod="kuadrant-system/authorino-58458bb776-79rkb" Sep 16 07:46:30.239095 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.239070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpgkf\" (UniqueName: \"kubernetes.io/projected/df2f5e81-9bba-4cd7-afdf-956188daa9ed-kube-api-access-fpgkf\") pod \"authorino-58458bb776-79rkb\" (UID: \"df2f5e81-9bba-4cd7-afdf-956188daa9ed\") " pod="kuadrant-system/authorino-58458bb776-79rkb" Sep 16 07:46:30.239197 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.239104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df2f5e81-9bba-4cd7-afdf-956188daa9ed-tls-cert\") pod \"authorino-58458bb776-79rkb\" (UID: \"df2f5e81-9bba-4cd7-afdf-956188daa9ed\") " pod="kuadrant-system/authorino-58458bb776-79rkb" Sep 16 07:46:30.241660 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.241632 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df2f5e81-9bba-4cd7-afdf-956188daa9ed-tls-cert\") pod \"authorino-58458bb776-79rkb\" (UID: \"df2f5e81-9bba-4cd7-afdf-956188daa9ed\") " pod="kuadrant-system/authorino-58458bb776-79rkb" Sep 16 07:46:30.248037 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.248017 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpgkf\" (UniqueName: \"kubernetes.io/projected/df2f5e81-9bba-4cd7-afdf-956188daa9ed-kube-api-access-fpgkf\") pod \"authorino-58458bb776-79rkb\" (UID: \"df2f5e81-9bba-4cd7-afdf-956188daa9ed\") " pod="kuadrant-system/authorino-58458bb776-79rkb" Sep 16 07:46:30.378113 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.378058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58458bb776-79rkb" Sep 16 07:46:30.649025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:30.649002 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58458bb776-79rkb"] Sep 16 07:46:30.650714 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:46:30.650686 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf2f5e81_9bba_4cd7_afdf_956188daa9ed.slice/crio-e985fc637702dccbb4bbbf76092ca1af88b1a162ae0a7c329472f4cb2557e6a6 WatchSource:0}: Error finding container e985fc637702dccbb4bbbf76092ca1af88b1a162ae0a7c329472f4cb2557e6a6: Status 404 returned error can't find the container with id e985fc637702dccbb4bbbf76092ca1af88b1a162ae0a7c329472f4cb2557e6a6 Sep 16 07:46:31.567284 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:31.567246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58458bb776-79rkb" event={"ID":"df2f5e81-9bba-4cd7-afdf-956188daa9ed","Type":"ContainerStarted","Data":"e985fc637702dccbb4bbbf76092ca1af88b1a162ae0a7c329472f4cb2557e6a6"} Sep 16 07:46:40.618104 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:40.618003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58458bb776-79rkb" event={"ID":"df2f5e81-9bba-4cd7-afdf-956188daa9ed","Type":"ContainerStarted","Data":"3c815c1500fcc8bebbb1bc5a4a5437faf6df94e53ba98dc6040f1bd43de15147"} Sep 16 07:46:40.639970 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:46:40.639913 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58458bb776-79rkb" podStartSLOduration=1.400936774 podStartE2EDuration="10.639893958s" totalImagesPullingTime="9.238957184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:46:40.639619026 +0000 UTC m=+773.559234974" watchObservedRunningTime="2026-09-16 07:46:40.639893958 +0000 UTC m=+773.559509905" Sep 16 07:47:02.121249 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:02.121196 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-73b4cf"] Sep 16 07:47:02.125326 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:02.125306 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-73b4cf" Sep 16 07:47:02.129566 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:02.127669 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kvkmt\"" Sep 16 07:47:02.136228 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:02.136182 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-73b4cf"] Sep 16 07:47:02.194047 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:02.194018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbj5f\" (UniqueName: \"kubernetes.io/projected/6a38702a-c7f3-47c8-ac72-64a2d4623b98-kube-api-access-vbj5f\") pod \"test-curl-7705-73b4cf\" (UID: \"6a38702a-c7f3-47c8-ac72-64a2d4623b98\") " pod="opendatahub/test-curl-7705-73b4cf" Sep 16 07:47:02.295089 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:02.295061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbj5f\" (UniqueName: \"kubernetes.io/projected/6a38702a-c7f3-47c8-ac72-64a2d4623b98-kube-api-access-vbj5f\") pod \"test-curl-7705-73b4cf\" (UID: \"6a38702a-c7f3-47c8-ac72-64a2d4623b98\") " pod="opendatahub/test-curl-7705-73b4cf" Sep 16 07:47:02.303927 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:02.303896 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbj5f\" (UniqueName: \"kubernetes.io/projected/6a38702a-c7f3-47c8-ac72-64a2d4623b98-kube-api-access-vbj5f\") pod \"test-curl-7705-73b4cf\" (UID: \"6a38702a-c7f3-47c8-ac72-64a2d4623b98\") " pod="opendatahub/test-curl-7705-73b4cf" Sep 16 07:47:02.443657 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:02.443633 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-73b4cf" Sep 16 07:47:02.720988 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:02.720770 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-73b4cf"] Sep 16 07:47:02.727618 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:47:02.727590 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a38702a_c7f3_47c8_ac72_64a2d4623b98.slice/crio-b9bb04a11ba90cf83b2489445ba255d3b1bfcdc2db95317f3ca88cf91ada8ffc WatchSource:0}: Error finding container b9bb04a11ba90cf83b2489445ba255d3b1bfcdc2db95317f3ca88cf91ada8ffc: Status 404 returned error can't find the container with id b9bb04a11ba90cf83b2489445ba255d3b1bfcdc2db95317f3ca88cf91ada8ffc Sep 16 07:47:02.741855 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:02.741825 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-73b4cf" event={"ID":"6a38702a-c7f3-47c8-ac72-64a2d4623b98","Type":"ContainerStarted","Data":"b9bb04a11ba90cf83b2489445ba255d3b1bfcdc2db95317f3ca88cf91ada8ffc"} Sep 16 07:47:03.010898 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:03.010804 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:47:05.266728 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.266696 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697"] Sep 16 07:47:05.270083 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.270065 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.272919 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.272899 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-c9020d-serving-cert\"" Sep 16 07:47:05.283819 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.283795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697"] Sep 16 07:47:05.322819 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.322793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.322955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.322830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqds5\" (UniqueName: \"kubernetes.io/projected/4557169e-72cc-4a7d-b42f-e87a5c1608a3-kube-api-access-kqds5\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.322955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.322866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4557169e-72cc-4a7d-b42f-e87a5c1608a3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.322955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.322931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-metrics-certs\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.424323 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.424292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-metrics-certs\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.424499 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.424404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.424499 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.424440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqds5\" (UniqueName: \"kubernetes.io/projected/4557169e-72cc-4a7d-b42f-e87a5c1608a3-kube-api-access-kqds5\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.424623 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.424588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4557169e-72cc-4a7d-b42f-e87a5c1608a3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.425307 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.425278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4557169e-72cc-4a7d-b42f-e87a5c1608a3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.426988 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.426965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.427105 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.426965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-metrics-certs\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.432812 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.432787 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqds5\" (UniqueName: \"kubernetes.io/projected/4557169e-72cc-4a7d-b42f-e87a5c1608a3-kube-api-access-kqds5\") pod \"maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.581707 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.581637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:05.765780 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.765742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-73b4cf" event={"ID":"6a38702a-c7f3-47c8-ac72-64a2d4623b98","Type":"ContainerStarted","Data":"1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a"} Sep 16 07:47:05.781936 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.781886 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7705-73b4cf" podStartSLOduration=1.7997724929999999 podStartE2EDuration="3.781871802s" totalImagesPullingTime="1.982099309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:05.781629914 +0000 UTC m=+798.701245859" watchObservedRunningTime="2026-09-16 07:47:05.781871802 +0000 UTC m=+798.701487767" Sep 16 07:47:05.868963 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:05.868933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697"] Sep 16 07:47:05.869542 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:47:05.869512 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4557169e_72cc_4a7d_b42f_e87a5c1608a3.slice/crio-9644efe183df8bb85df5efa5d9818804a53084eb4fd9750dd521eb4a2406d6a2 WatchSource:0}: Error finding container 9644efe183df8bb85df5efa5d9818804a53084eb4fd9750dd521eb4a2406d6a2: Status 404 returned error can't find the container with id 9644efe183df8bb85df5efa5d9818804a53084eb4fd9750dd521eb4a2406d6a2 Sep 16 07:47:06.099610 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.099564 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2"] Sep 16 07:47:06.104475 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.104454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:47:06.107597 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.107574 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c9020d-dockercfg-tbl6d\"" Sep 16 07:47:06.107991 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.107976 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c9020d\"" Sep 16 07:47:06.114147 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.114124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2"] Sep 16 07:47:06.231019 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.230986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nvtl\" (UniqueName: \"kubernetes.io/projected/79178c36-2949-484f-b547-aed1a748fd50-kube-api-access-5nvtl\") pod \"payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2\" (UID: \"79178c36-2949-484f-b547-aed1a748fd50\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:47:06.231174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.231052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79178c36-2949-484f-b547-aed1a748fd50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2\" (UID: \"79178c36-2949-484f-b547-aed1a748fd50\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:47:06.332372 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.332344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nvtl\" (UniqueName: \"kubernetes.io/projected/79178c36-2949-484f-b547-aed1a748fd50-kube-api-access-5nvtl\") pod \"payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2\" (UID: \"79178c36-2949-484f-b547-aed1a748fd50\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:47:06.332744 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.332402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79178c36-2949-484f-b547-aed1a748fd50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2\" (UID: \"79178c36-2949-484f-b547-aed1a748fd50\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:47:06.333034 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.333010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79178c36-2949-484f-b547-aed1a748fd50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2\" (UID: \"79178c36-2949-484f-b547-aed1a748fd50\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:47:06.340656 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.340629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nvtl\" (UniqueName: \"kubernetes.io/projected/79178c36-2949-484f-b547-aed1a748fd50-kube-api-access-5nvtl\") pod \"payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2\" (UID: \"79178c36-2949-484f-b547-aed1a748fd50\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:47:06.363348 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.363310 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-73b4cf"] Sep 16 07:47:06.366788 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.366716 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-73b4cf"] Sep 16 07:47:06.417186 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.417105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:47:06.482523 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.482491 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-f16c9c"] Sep 16 07:47:06.483150 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.483133 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a38702a-c7f3-47c8-ac72-64a2d4623b98" containerName="test-curl-7705-73b4cf" Sep 16 07:47:06.483202 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.483155 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a38702a-c7f3-47c8-ac72-64a2d4623b98" containerName="test-curl-7705-73b4cf" Sep 16 07:47:06.483273 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.483262 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a38702a-c7f3-47c8-ac72-64a2d4623b98" containerName="test-curl-7705-73b4cf" Sep 16 07:47:06.488100 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.488075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-f16c9c" Sep 16 07:47:06.492454 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.492430 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-f16c9c"] Sep 16 07:47:06.534352 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.534324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l529j\" (UniqueName: \"kubernetes.io/projected/631f7f77-39ab-4f4a-abf1-a43ce9ebf533-kube-api-access-l529j\") pod \"test-curl-7705-f16c9c\" (UID: \"631f7f77-39ab-4f4a-abf1-a43ce9ebf533\") " pod="opendatahub/test-curl-7705-f16c9c" Sep 16 07:47:06.635841 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.635802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l529j\" (UniqueName: \"kubernetes.io/projected/631f7f77-39ab-4f4a-abf1-a43ce9ebf533-kube-api-access-l529j\") pod \"test-curl-7705-f16c9c\" (UID: \"631f7f77-39ab-4f4a-abf1-a43ce9ebf533\") " pod="opendatahub/test-curl-7705-f16c9c" Sep 16 07:47:06.644232 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.644181 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l529j\" (UniqueName: \"kubernetes.io/projected/631f7f77-39ab-4f4a-abf1-a43ce9ebf533-kube-api-access-l529j\") pod \"test-curl-7705-f16c9c\" (UID: \"631f7f77-39ab-4f4a-abf1-a43ce9ebf533\") " pod="opendatahub/test-curl-7705-f16c9c" Sep 16 07:47:06.690344 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.690318 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2"] Sep 16 07:47:06.691644 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:47:06.691619 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79178c36_2949_484f_b547_aed1a748fd50.slice/crio-19b83d247ce16f4cce107e6c3f4f617d6a9a86ac1e56981376c9ea45e7ffb64f WatchSource:0}: Error finding container 19b83d247ce16f4cce107e6c3f4f617d6a9a86ac1e56981376c9ea45e7ffb64f: Status 404 returned error can't find the container with id 19b83d247ce16f4cce107e6c3f4f617d6a9a86ac1e56981376c9ea45e7ffb64f Sep 16 07:47:06.773828 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.773789 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" event={"ID":"4557169e-72cc-4a7d-b42f-e87a5c1608a3","Type":"ContainerStarted","Data":"d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd"} Sep 16 07:47:06.774013 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.773861 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" event={"ID":"4557169e-72cc-4a7d-b42f-e87a5c1608a3","Type":"ContainerStarted","Data":"9644efe183df8bb85df5efa5d9818804a53084eb4fd9750dd521eb4a2406d6a2"} Sep 16 07:47:06.774013 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.773975 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:06.775901 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.775858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" event={"ID":"79178c36-2949-484f-b547-aed1a748fd50","Type":"ContainerStarted","Data":"ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4"} Sep 16 07:47:06.776063 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.775922 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" event={"ID":"79178c36-2949-484f-b547-aed1a748fd50","Type":"ContainerStarted","Data":"19b83d247ce16f4cce107e6c3f4f617d6a9a86ac1e56981376c9ea45e7ffb64f"} Sep 16 07:47:06.776063 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.775964 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:47:06.776237 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:06.776082 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7705-73b4cf" container="test-curl-7705-73b4cf" Sep 16 07:47:06.800738 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.800706 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-f16c9c" Sep 16 07:47:06.807087 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.807016 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" podStartSLOduration=1.390205949 podStartE2EDuration="1.806996128s" totalImagesPullingTime="416.790179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:06.80643614 +0000 UTC m=+799.726052087" watchObservedRunningTime="2026-09-16 07:47:06.806996128 +0000 UTC m=+799.726612077" Sep 16 07:47:06.845151 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:06.844421 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" podStartSLOduration=0.844397379 podStartE2EDuration="844.397379ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:06.840757532 +0000 UTC m=+799.760373479" watchObservedRunningTime="2026-09-16 07:47:06.844397379 +0000 UTC m=+799.764013328" Sep 16 07:47:07.077846 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.077814 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-f16c9c"] Sep 16 07:47:07.079197 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:47:07.079163 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod631f7f77_39ab_4f4a_abf1_a43ce9ebf533.slice/crio-728394c8a6f52bb84bd3a2687e0c3283cca0c983051e8f0ef3524171d80b41d6 WatchSource:0}: Error finding container 728394c8a6f52bb84bd3a2687e0c3283cca0c983051e8f0ef3524171d80b41d6: Status 404 returned error can't find the container with id 728394c8a6f52bb84bd3a2687e0c3283cca0c983051e8f0ef3524171d80b41d6 Sep 16 07:47:07.355874 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.355787 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj"] Sep 16 07:47:07.359795 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.359772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.362395 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.362373 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5dd30f-serving-cert\"" Sep 16 07:47:07.370918 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.370892 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj"] Sep 16 07:47:07.443464 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.443433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92611524-dc02-4c2f-a048-ce8570aaada8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.443621 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.443484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-maas-api-tls\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.443621 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.443549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-metrics-certs\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.443621 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.443610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc9dp\" (UniqueName: \"kubernetes.io/projected/92611524-dc02-4c2f-a048-ce8570aaada8-kube-api-access-pc9dp\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.545127 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.545096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-maas-api-tls\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.545302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.545142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-metrics-certs\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.545302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.545181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc9dp\" (UniqueName: \"kubernetes.io/projected/92611524-dc02-4c2f-a048-ce8570aaada8-kube-api-access-pc9dp\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.545302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.545237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92611524-dc02-4c2f-a048-ce8570aaada8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.545471 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:07.545306 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-serving-cert: secret "maas-api-e2e-inf-a-5dd30f-serving-cert" not found Sep 16 07:47:07.545471 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:07.545397 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-maas-api-tls podName:92611524-dc02-4c2f-a048-ce8570aaada8 nodeName:}" failed. No retries permitted until 2026-09-16 07:47:08.045374552 +0000 UTC m=+800.964990492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-maas-api-tls") pod "maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" (UID: "92611524-dc02-4c2f-a048-ce8570aaada8") : secret "maas-api-e2e-inf-a-5dd30f-serving-cert" not found Sep 16 07:47:07.545922 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.545905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92611524-dc02-4c2f-a048-ce8570aaada8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.547747 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.547728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-metrics-certs\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.557480 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.557458 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc9dp\" (UniqueName: \"kubernetes.io/projected/92611524-dc02-4c2f-a048-ce8570aaada8-kube-api-access-pc9dp\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:07.783350 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.783311 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-f16c9c" event={"ID":"631f7f77-39ab-4f4a-abf1-a43ce9ebf533","Type":"ContainerStarted","Data":"872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5"} Sep 16 07:47:07.783521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.783361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-f16c9c" event={"ID":"631f7f77-39ab-4f4a-abf1-a43ce9ebf533","Type":"ContainerStarted","Data":"728394c8a6f52bb84bd3a2687e0c3283cca0c983051e8f0ef3524171d80b41d6"} Sep 16 07:47:07.783660 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.783631 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7705-73b4cf" podUID="6a38702a-c7f3-47c8-ac72-64a2d4623b98" containerName="test-curl-7705-73b4cf" containerID="cri-o://1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a" gracePeriod=2 Sep 16 07:47:07.799553 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:07.799503 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7705-f16c9c" podStartSLOduration=1.7994885360000001 podStartE2EDuration="1.799488536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:07.79896042 +0000 UTC m=+800.718576369" watchObservedRunningTime="2026-09-16 07:47:07.799488536 +0000 UTC m=+800.719104481" Sep 16 07:47:08.049647 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.049564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-maas-api-tls\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:08.052058 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.052030 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-maas-api-tls\") pod \"maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:08.272921 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.272878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:08.415749 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.415718 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-f16c9c"] Sep 16 07:47:08.421885 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.421854 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-f16c9c"] Sep 16 07:47:08.443619 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.443590 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks"] Sep 16 07:47:08.444356 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.444199 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631f7f77-39ab-4f4a-abf1-a43ce9ebf533" containerName="test-curl-7705-f16c9c" Sep 16 07:47:08.444356 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.444236 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="631f7f77-39ab-4f4a-abf1-a43ce9ebf533" containerName="test-curl-7705-f16c9c" Sep 16 07:47:08.444539 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.444435 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="631f7f77-39ab-4f4a-abf1-a43ce9ebf533" containerName="test-curl-7705-f16c9c" Sep 16 07:47:08.448878 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.448856 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.452303 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.452273 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-fc27a5-serving-cert\"" Sep 16 07:47:08.461747 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.461720 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks"] Sep 16 07:47:08.551500 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.551477 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj"] Sep 16 07:47:08.553282 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:47:08.553251 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92611524_dc02_4c2f_a048_ce8570aaada8.slice/crio-e7b3394b1a170f7fab8656371bd288dec2afff349a8e25e1ddb96bbe43035004 WatchSource:0}: Error finding container e7b3394b1a170f7fab8656371bd288dec2afff349a8e25e1ddb96bbe43035004: Status 404 returned error can't find the container with id e7b3394b1a170f7fab8656371bd288dec2afff349a8e25e1ddb96bbe43035004 Sep 16 07:47:08.553433 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.553416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-metrics-certs\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.553479 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.553458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bzgz\" (UniqueName: \"kubernetes.io/projected/6d80d240-6d72-4abb-ab72-69992b317fa2-kube-api-access-5bzgz\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.553557 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.553521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d80d240-6d72-4abb-ab72-69992b317fa2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.553674 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.553657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.654364 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.654337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bzgz\" (UniqueName: \"kubernetes.io/projected/6d80d240-6d72-4abb-ab72-69992b317fa2-kube-api-access-5bzgz\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.654485 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.654370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d80d240-6d72-4abb-ab72-69992b317fa2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.654485 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.654418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.654485 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.654457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-metrics-certs\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.655132 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.655109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d80d240-6d72-4abb-ab72-69992b317fa2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.656814 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.656796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-metrics-certs\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.656982 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.656956 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.663291 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.663272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bzgz\" (UniqueName: \"kubernetes.io/projected/6d80d240-6d72-4abb-ab72-69992b317fa2-kube-api-access-5bzgz\") pod \"maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.762668 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.762638 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:08.792473 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:08.792443 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" event={"ID":"92611524-dc02-4c2f-a048-ce8570aaada8","Type":"ContainerStarted","Data":"e7b3394b1a170f7fab8656371bd288dec2afff349a8e25e1ddb96bbe43035004"} Sep 16 07:47:08.792754 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:08.792734 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7705-f16c9c" container="test-curl-7705-f16c9c" Sep 16 07:47:09.051081 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.051054 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks"] Sep 16 07:47:09.051851 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:47:09.051828 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d80d240_6d72_4abb_ab72_69992b317fa2.slice/crio-379f5a04f12a8036d540dd2b26ea475d6236464cfd44a9922aa07e88ac07af6a WatchSource:0}: Error finding container 379f5a04f12a8036d540dd2b26ea475d6236464cfd44a9922aa07e88ac07af6a: Status 404 returned error can't find the container with id 379f5a04f12a8036d540dd2b26ea475d6236464cfd44a9922aa07e88ac07af6a Sep 16 07:47:09.799893 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.799869 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" event={"ID":"92611524-dc02-4c2f-a048-ce8570aaada8","Type":"ContainerStarted","Data":"2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39"} Sep 16 07:47:09.800290 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.799931 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:09.801481 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.801457 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" event={"ID":"6d80d240-6d72-4abb-ab72-69992b317fa2","Type":"ContainerStarted","Data":"caf0988c2f0f785edfbf5a66b94f034f722a380efc44a552f18c583d146fe110"} Sep 16 07:47:09.801585 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.801499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" event={"ID":"6d80d240-6d72-4abb-ab72-69992b317fa2","Type":"ContainerStarted","Data":"379f5a04f12a8036d540dd2b26ea475d6236464cfd44a9922aa07e88ac07af6a"} Sep 16 07:47:09.801585 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.801518 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:09.801585 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.801537 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7705-f16c9c" podUID="631f7f77-39ab-4f4a-abf1-a43ce9ebf533" containerName="test-curl-7705-f16c9c" containerID="cri-o://872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5" gracePeriod=2 Sep 16 07:47:09.819702 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.819654 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" podStartSLOduration=2.377739752 podStartE2EDuration="2.819639663s" totalImagesPullingTime="441.899911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:09.816669802 +0000 UTC m=+802.736285750" watchObservedRunningTime="2026-09-16 07:47:09.819639663 +0000 UTC m=+802.739255608" Sep 16 07:47:09.836660 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.836598 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" podStartSLOduration=1.497793611 podStartE2EDuration="1.836576601s" totalImagesPullingTime="338.78299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:09.835139065 +0000 UTC m=+802.754755013" watchObservedRunningTime="2026-09-16 07:47:09.836576601 +0000 UTC m=+802.756192549" Sep 16 07:47:09.929955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.929931 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-73b4cf" Sep 16 07:47:09.967153 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.967128 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbj5f\" (UniqueName: \"kubernetes.io/projected/6a38702a-c7f3-47c8-ac72-64a2d4623b98-kube-api-access-vbj5f\") pod \"6a38702a-c7f3-47c8-ac72-64a2d4623b98\" (UID: \"6a38702a-c7f3-47c8-ac72-64a2d4623b98\") " Sep 16 07:47:09.969121 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:09.969093 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a38702a-c7f3-47c8-ac72-64a2d4623b98-kube-api-access-vbj5f" (OuterVolumeSpecName: "kube-api-access-vbj5f") pod "6a38702a-c7f3-47c8-ac72-64a2d4623b98" (UID: "6a38702a-c7f3-47c8-ac72-64a2d4623b98"). InnerVolumeSpecName "kube-api-access-vbj5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:47:10.068668 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:10.068648 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbj5f\" (UniqueName: \"kubernetes.io/projected/6a38702a-c7f3-47c8-ac72-64a2d4623b98-kube-api-access-vbj5f\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:47:10.813948 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:10.813901 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a38702a-c7f3-47c8-ac72-64a2d4623b98" containerID="1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a" exitCode=137 Sep 16 07:47:10.814442 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:10.814097 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-73b4cf" Sep 16 07:47:10.814442 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:10.814184 2580 scope.go:117] "RemoveContainer" containerID="1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a" Sep 16 07:47:10.826418 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:10.826171 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-73b4cf\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="6a38702a-c7f3-47c8-ac72-64a2d4623b98" pod="opendatahub/test-curl-7705-73b4cf" Sep 16 07:47:10.826418 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:10.826353 2580 scope.go:117] "RemoveContainer" containerID="1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a" Sep 16 07:47:10.826664 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:10.826643 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a\": container with ID starting with 1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a not found: ID does not exist" containerID="1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a" Sep 16 07:47:10.826750 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:10.826671 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a"} err="failed to get container status \"1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a\": rpc error: code = NotFound desc = could not find container \"1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a\": container with ID starting with 1ba2156771f3e0c81982721fdcee600ae502a079ae7d31b1835816bdf71ded7a not found: ID does not exist" Sep 16 07:47:11.561278 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:11.561235 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a38702a-c7f3-47c8-ac72-64a2d4623b98" path="/var/lib/kubelet/pods/6a38702a-c7f3-47c8-ac72-64a2d4623b98/volumes" Sep 16 07:47:11.982294 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:11.982266 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-f16c9c" Sep 16 07:47:12.089253 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:12.089150 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l529j\" (UniqueName: \"kubernetes.io/projected/631f7f77-39ab-4f4a-abf1-a43ce9ebf533-kube-api-access-l529j\") pod \"631f7f77-39ab-4f4a-abf1-a43ce9ebf533\" (UID: \"631f7f77-39ab-4f4a-abf1-a43ce9ebf533\") " Sep 16 07:47:12.091219 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:12.091181 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631f7f77-39ab-4f4a-abf1-a43ce9ebf533-kube-api-access-l529j" (OuterVolumeSpecName: "kube-api-access-l529j") pod "631f7f77-39ab-4f4a-abf1-a43ce9ebf533" (UID: "631f7f77-39ab-4f4a-abf1-a43ce9ebf533"). InnerVolumeSpecName "kube-api-access-l529j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:47:12.190328 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:12.190299 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l529j\" (UniqueName: \"kubernetes.io/projected/631f7f77-39ab-4f4a-abf1-a43ce9ebf533-kube-api-access-l529j\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:47:12.788701 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:12.788673 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:47:12.832430 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:12.832400 2580 generic.go:356] "Generic (PLEG): container finished" podID="631f7f77-39ab-4f4a-abf1-a43ce9ebf533" containerID="872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5" exitCode=137 Sep 16 07:47:12.832607 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:12.832437 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-f16c9c" Sep 16 07:47:12.832607 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:12.832501 2580 scope.go:117] "RemoveContainer" containerID="872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5" Sep 16 07:47:12.843307 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:12.843064 2580 scope.go:117] "RemoveContainer" containerID="872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5" Sep 16 07:47:12.843406 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:12.843384 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5\": container with ID starting with 872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5 not found: ID does not exist" containerID="872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5" Sep 16 07:47:12.843510 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:12.843414 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5"} err="failed to get container status \"872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5\": rpc error: code = NotFound desc = could not find container \"872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5\": container with ID starting with 872ea48714eae75bf9668b03c9e5419846148ac1054c7e5c546d5f600c0a39c5 not found: ID does not exist" Sep 16 07:47:12.843938 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:12.843909 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-f16c9c\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="631f7f77-39ab-4f4a-abf1-a43ce9ebf533" pod="opendatahub/test-curl-7705-f16c9c" Sep 16 07:47:13.560884 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:13.560853 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="631f7f77-39ab-4f4a-abf1-a43ce9ebf533" path="/var/lib/kubelet/pods/631f7f77-39ab-4f4a-abf1-a43ce9ebf533/volumes" Sep 16 07:47:14.936391 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:14.936356 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk"] Sep 16 07:47:14.941156 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:14.941134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:14.944302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:14.944277 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0de2c4d3-serving-cert\"" Sep 16 07:47:14.948668 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:14.948647 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk"] Sep 16 07:47:15.017564 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.017526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-maas-api-tls\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.017714 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.017588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-metrics-certs\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.017714 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.017698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s26j\" (UniqueName: \"kubernetes.io/projected/3886238a-4f8a-4cbc-9377-70b065a71e65-kube-api-access-2s26j\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.017806 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.017735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3886238a-4f8a-4cbc-9377-70b065a71e65-openshift-service-ca\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.118717 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.118689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-maas-api-tls\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.118859 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.118724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-metrics-certs\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.118859 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.118764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s26j\" (UniqueName: \"kubernetes.io/projected/3886238a-4f8a-4cbc-9377-70b065a71e65-kube-api-access-2s26j\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.118859 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.118787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3886238a-4f8a-4cbc-9377-70b065a71e65-openshift-service-ca\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.119424 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.119403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3886238a-4f8a-4cbc-9377-70b065a71e65-openshift-service-ca\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.121433 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.121403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-maas-api-tls\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.121534 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.121483 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-metrics-certs\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.130723 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.130698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s26j\" (UniqueName: \"kubernetes.io/projected/3886238a-4f8a-4cbc-9377-70b065a71e65-kube-api-access-2s26j\") pod \"maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.253703 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.253628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:15.553265 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.553235 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk"] Sep 16 07:47:15.554830 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:47:15.554803 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3886238a_4f8a_4cbc_9377_70b065a71e65.slice/crio-3a4e2164342692b927c137e036c624e15f906cf9d26d3e9be945eff23cc6474e WatchSource:0}: Error finding container 3a4e2164342692b927c137e036c624e15f906cf9d26d3e9be945eff23cc6474e: Status 404 returned error can't find the container with id 3a4e2164342692b927c137e036c624e15f906cf9d26d3e9be945eff23cc6474e Sep 16 07:47:15.819983 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.819902 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 07:47:15.820466 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.820446 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:47:15.866476 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:15.866369 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" event={"ID":"3886238a-4f8a-4cbc-9377-70b065a71e65","Type":"ContainerStarted","Data":"3a4e2164342692b927c137e036c624e15f906cf9d26d3e9be945eff23cc6474e"} Sep 16 07:47:16.874947 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:16.874914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" event={"ID":"3886238a-4f8a-4cbc-9377-70b065a71e65","Type":"ContainerStarted","Data":"91bd495a258b35abe28f320eb564e71c0cf6c4da52df04d8d91db38cff48a434"} Sep 16 07:47:16.875363 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:16.875151 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:16.893972 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:16.893927 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" podStartSLOduration=2.520012033 podStartE2EDuration="2.893909197s" totalImagesPullingTime="373.897164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:16.891848155 +0000 UTC m=+809.811464115" watchObservedRunningTime="2026-09-16 07:47:16.893909197 +0000 UTC m=+809.813525144" Sep 16 07:47:17.785933 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:17.785902 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:47:18.486864 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.486824 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb"] Sep 16 07:47:18.490489 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.490473 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.492976 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.492946 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-b60434-serving-cert\"" Sep 16 07:47:18.502950 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.502924 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb"] Sep 16 07:47:18.652293 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.652255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-maas-api-tls\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.652483 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.652315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhvcv\" (UniqueName: \"kubernetes.io/projected/5f876c73-a707-4772-9dcb-b6bcf412cc90-kube-api-access-jhvcv\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.652483 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.652349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-metrics-certs\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.652483 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.652415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f876c73-a707-4772-9dcb-b6bcf412cc90-openshift-service-ca\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.753198 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.753131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-maas-api-tls\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.753198 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.753174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvcv\" (UniqueName: \"kubernetes.io/projected/5f876c73-a707-4772-9dcb-b6bcf412cc90-kube-api-access-jhvcv\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.753198 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.753197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-metrics-certs\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.753383 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.753248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f876c73-a707-4772-9dcb-b6bcf412cc90-openshift-service-ca\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.753383 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:18.753312 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-b60434-serving-cert: secret "maas-api-e2e-autores-b60434-serving-cert" not found Sep 16 07:47:18.753383 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:18.753375 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-maas-api-tls podName:5f876c73-a707-4772-9dcb-b6bcf412cc90 nodeName:}" failed. No retries permitted until 2026-09-16 07:47:19.25336022 +0000 UTC m=+812.172976143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-maas-api-tls") pod "maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" (UID: "5f876c73-a707-4772-9dcb-b6bcf412cc90") : secret "maas-api-e2e-autores-b60434-serving-cert" not found Sep 16 07:47:18.753861 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.753841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f876c73-a707-4772-9dcb-b6bcf412cc90-openshift-service-ca\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.755758 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.755735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-metrics-certs\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.764632 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.764611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvcv\" (UniqueName: \"kubernetes.io/projected/5f876c73-a707-4772-9dcb-b6bcf412cc90-kube-api-access-jhvcv\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:18.911804 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:18.911770 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:47:19.257503 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:19.257468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-maas-api-tls\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:19.260087 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:19.260051 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-maas-api-tls\") pod \"maas-api-e2e-autores-b60434-5c9f97dc45-rjttb\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:19.402840 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:19.402806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:19.677065 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:19.677032 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb"] Sep 16 07:47:19.681348 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:47:19.681312 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f876c73_a707_4772_9dcb_b6bcf412cc90.slice/crio-c9b61a5f87a28d74e9a7f0f90824ebbf95e5c3199a9faea6abe2976a6de2c9df WatchSource:0}: Error finding container c9b61a5f87a28d74e9a7f0f90824ebbf95e5c3199a9faea6abe2976a6de2c9df: Status 404 returned error can't find the container with id c9b61a5f87a28d74e9a7f0f90824ebbf95e5c3199a9faea6abe2976a6de2c9df Sep 16 07:47:19.898074 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:19.898027 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" event={"ID":"5f876c73-a707-4772-9dcb-b6bcf412cc90","Type":"ContainerStarted","Data":"c9b61a5f87a28d74e9a7f0f90824ebbf95e5c3199a9faea6abe2976a6de2c9df"} Sep 16 07:47:20.905466 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:20.905429 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" event={"ID":"5f876c73-a707-4772-9dcb-b6bcf412cc90","Type":"ContainerStarted","Data":"a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6"} Sep 16 07:47:20.905884 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:20.905497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:20.927722 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:20.927671 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" podStartSLOduration=2.197386684 podStartE2EDuration="2.927657027s" totalImagesPullingTime="730.270343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:20.924071412 +0000 UTC m=+813.843687380" watchObservedRunningTime="2026-09-16 07:47:20.927657027 +0000 UTC m=+813.847272982" Sep 16 07:47:22.888604 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:22.888575 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:26.919230 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:26.919177 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:47:32.554350 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.554320 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz"] Sep 16 07:47:32.558170 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.558152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.560830 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.560810 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dd359c-serving-cert\"" Sep 16 07:47:32.571691 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.571666 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz"] Sep 16 07:47:32.679570 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.679541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.679720 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.679574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-metrics-certs\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.679720 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.679598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e6987a-0fa0-48ad-8101-9fc027c576e6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.679797 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.679719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68f9\" (UniqueName: \"kubernetes.io/projected/d2e6987a-0fa0-48ad-8101-9fc027c576e6-kube-api-access-z68f9\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.780834 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.780800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68f9\" (UniqueName: \"kubernetes.io/projected/d2e6987a-0fa0-48ad-8101-9fc027c576e6-kube-api-access-z68f9\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.780989 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.780902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.780989 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.780933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-metrics-certs\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.780989 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.780960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e6987a-0fa0-48ad-8101-9fc027c576e6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.781161 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:32.781038 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-serving-cert: secret "maas-api-e2e-inf-b-dd359c-serving-cert" not found Sep 16 07:47:32.781161 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:47:32.781116 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-maas-api-tls podName:d2e6987a-0fa0-48ad-8101-9fc027c576e6 nodeName:}" failed. No retries permitted until 2026-09-16 07:47:33.281098678 +0000 UTC m=+826.200714619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-maas-api-tls") pod "maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" (UID: "d2e6987a-0fa0-48ad-8101-9fc027c576e6") : secret "maas-api-e2e-inf-b-dd359c-serving-cert" not found Sep 16 07:47:32.781689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.781666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e6987a-0fa0-48ad-8101-9fc027c576e6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.783475 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.783451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-metrics-certs\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:32.791746 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:32.791729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68f9\" (UniqueName: \"kubernetes.io/projected/d2e6987a-0fa0-48ad-8101-9fc027c576e6-kube-api-access-z68f9\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:33.285394 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:33.285358 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:33.287727 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:33.287697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:33.469459 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:33.469423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:33.743990 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:33.743962 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz"] Sep 16 07:47:33.744882 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:47:33.744854 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e6987a_0fa0_48ad_8101_9fc027c576e6.slice/crio-a37faea4f528b14f2ddf04d467415b2a90e4dcd29f28b12aeac34fe93b4228e7 WatchSource:0}: Error finding container a37faea4f528b14f2ddf04d467415b2a90e4dcd29f28b12aeac34fe93b4228e7: Status 404 returned error can't find the container with id a37faea4f528b14f2ddf04d467415b2a90e4dcd29f28b12aeac34fe93b4228e7 Sep 16 07:47:33.995275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:33.995189 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" event={"ID":"d2e6987a-0fa0-48ad-8101-9fc027c576e6","Type":"ContainerStarted","Data":"a37faea4f528b14f2ddf04d467415b2a90e4dcd29f28b12aeac34fe93b4228e7"} Sep 16 07:47:35.004292 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:35.004248 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" event={"ID":"d2e6987a-0fa0-48ad-8101-9fc027c576e6","Type":"ContainerStarted","Data":"0d89814f794c47f70041111caa69d076d476ff2afd08ec9aacecbadd15141ca0"} Sep 16 07:47:35.004753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:35.004434 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:35.023822 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:35.023778 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" podStartSLOduration=2.5814793700000003 podStartE2EDuration="3.023763479s" totalImagesPullingTime="442.284109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:35.021580944 +0000 UTC m=+827.941196891" watchObservedRunningTime="2026-09-16 07:47:35.023763479 +0000 UTC m=+827.943379425" Sep 16 07:47:38.113739 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:38.113703 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:47:41.015735 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:41.015705 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:47:42.823646 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:42.823610 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk"] Sep 16 07:47:42.824102 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:42.823927 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" podUID="3886238a-4f8a-4cbc-9377-70b065a71e65" containerName="maas-api" containerID="cri-o://91bd495a258b35abe28f320eb564e71c0cf6c4da52df04d8d91db38cff48a434" gracePeriod=30 Sep 16 07:47:42.884014 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:42.883971 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" podUID="3886238a-4f8a-4cbc-9377-70b065a71e65" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.65:8443/health\": dial tcp 10.132.0.65:8443: connect: connection refused" Sep 16 07:47:43.054188 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.054149 2580 generic.go:356] "Generic (PLEG): container finished" podID="3886238a-4f8a-4cbc-9377-70b065a71e65" containerID="91bd495a258b35abe28f320eb564e71c0cf6c4da52df04d8d91db38cff48a434" exitCode=0 Sep 16 07:47:43.054188 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.054166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" event={"ID":"3886238a-4f8a-4cbc-9377-70b065a71e65","Type":"ContainerDied","Data":"91bd495a258b35abe28f320eb564e71c0cf6c4da52df04d8d91db38cff48a434"} Sep 16 07:47:43.081942 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.081890 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:43.178434 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.178404 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-maas-api-tls\") pod \"3886238a-4f8a-4cbc-9377-70b065a71e65\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " Sep 16 07:47:43.178597 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.178490 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-metrics-certs\") pod \"3886238a-4f8a-4cbc-9377-70b065a71e65\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " Sep 16 07:47:43.178597 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.178534 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3886238a-4f8a-4cbc-9377-70b065a71e65-openshift-service-ca\") pod \"3886238a-4f8a-4cbc-9377-70b065a71e65\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " Sep 16 07:47:43.178597 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.178553 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s26j\" (UniqueName: \"kubernetes.io/projected/3886238a-4f8a-4cbc-9377-70b065a71e65-kube-api-access-2s26j\") pod \"3886238a-4f8a-4cbc-9377-70b065a71e65\" (UID: \"3886238a-4f8a-4cbc-9377-70b065a71e65\") " Sep 16 07:47:43.179615 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.179259 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3886238a-4f8a-4cbc-9377-70b065a71e65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3886238a-4f8a-4cbc-9377-70b065a71e65" (UID: "3886238a-4f8a-4cbc-9377-70b065a71e65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:47:43.180931 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.180879 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3886238a-4f8a-4cbc-9377-70b065a71e65" (UID: "3886238a-4f8a-4cbc-9377-70b065a71e65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:47:43.181064 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.180998 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3886238a-4f8a-4cbc-9377-70b065a71e65-kube-api-access-2s26j" (OuterVolumeSpecName: "kube-api-access-2s26j") pod "3886238a-4f8a-4cbc-9377-70b065a71e65" (UID: "3886238a-4f8a-4cbc-9377-70b065a71e65"). InnerVolumeSpecName "kube-api-access-2s26j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:47:43.181132 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.181085 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3886238a-4f8a-4cbc-9377-70b065a71e65" (UID: "3886238a-4f8a-4cbc-9377-70b065a71e65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:47:43.279784 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.279752 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:47:43.279784 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.279778 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3886238a-4f8a-4cbc-9377-70b065a71e65-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:47:43.279784 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.279789 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3886238a-4f8a-4cbc-9377-70b065a71e65-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:47:43.279993 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:43.279799 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s26j\" (UniqueName: \"kubernetes.io/projected/3886238a-4f8a-4cbc-9377-70b065a71e65-kube-api-access-2s26j\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:47:44.061832 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:44.061792 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" event={"ID":"3886238a-4f8a-4cbc-9377-70b065a71e65","Type":"ContainerDied","Data":"3a4e2164342692b927c137e036c624e15f906cf9d26d3e9be945eff23cc6474e"} Sep 16 07:47:44.062299 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:44.061861 2580 scope.go:117] "RemoveContainer" containerID="91bd495a258b35abe28f320eb564e71c0cf6c4da52df04d8d91db38cff48a434" Sep 16 07:47:44.062299 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:44.061802 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk" Sep 16 07:47:44.085779 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:44.085749 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk"] Sep 16 07:47:44.092158 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:44.092130 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0de2c4d3-c5d8f7898-kgtdk"] Sep 16 07:47:45.560715 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:45.560681 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3886238a-4f8a-4cbc-9377-70b065a71e65" path="/var/lib/kubelet/pods/3886238a-4f8a-4cbc-9377-70b065a71e65/volumes" Sep 16 07:47:50.356749 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.356716 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b"] Sep 16 07:47:50.357287 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.357269 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3886238a-4f8a-4cbc-9377-70b065a71e65" containerName="maas-api" Sep 16 07:47:50.357287 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.357288 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3886238a-4f8a-4cbc-9377-70b065a71e65" containerName="maas-api" Sep 16 07:47:50.357409 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.357379 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3886238a-4f8a-4cbc-9377-70b065a71e65" containerName="maas-api" Sep 16 07:47:50.362144 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.362123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.364881 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.364859 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e5fb973f-serving-cert\"" Sep 16 07:47:50.370676 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.370654 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b"] Sep 16 07:47:50.445045 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.445020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-maas-api-tls\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.445159 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.445053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60025a6-82c1-494e-bb90-d7880e88fbe4-openshift-service-ca\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.445159 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.445082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh85r\" (UniqueName: \"kubernetes.io/projected/e60025a6-82c1-494e-bb90-d7880e88fbe4-kube-api-access-xh85r\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.445281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.445173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-metrics-certs\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.546464 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.546427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-maas-api-tls\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.546624 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.546499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60025a6-82c1-494e-bb90-d7880e88fbe4-openshift-service-ca\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.546624 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.546561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh85r\" (UniqueName: \"kubernetes.io/projected/e60025a6-82c1-494e-bb90-d7880e88fbe4-kube-api-access-xh85r\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.546624 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.546618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-metrics-certs\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.547302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.547274 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60025a6-82c1-494e-bb90-d7880e88fbe4-openshift-service-ca\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.548879 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.548850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-metrics-certs\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.548977 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.548900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-maas-api-tls\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.554313 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.554294 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh85r\" (UniqueName: \"kubernetes.io/projected/e60025a6-82c1-494e-bb90-d7880e88fbe4-kube-api-access-xh85r\") pod \"maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.675365 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.675331 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:50.951227 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:50.951184 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b"] Sep 16 07:47:50.951742 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:47:50.951712 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60025a6_82c1_494e_bb90_d7880e88fbe4.slice/crio-ea56a9007d85776547659605cde3b85380410e6b382bd4709889f7d50df5fd58 WatchSource:0}: Error finding container ea56a9007d85776547659605cde3b85380410e6b382bd4709889f7d50df5fd58: Status 404 returned error can't find the container with id ea56a9007d85776547659605cde3b85380410e6b382bd4709889f7d50df5fd58 Sep 16 07:47:51.115661 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:51.115623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" event={"ID":"e60025a6-82c1-494e-bb90-d7880e88fbe4","Type":"ContainerStarted","Data":"ea56a9007d85776547659605cde3b85380410e6b382bd4709889f7d50df5fd58"} Sep 16 07:47:52.123456 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:52.123416 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" event={"ID":"e60025a6-82c1-494e-bb90-d7880e88fbe4","Type":"ContainerStarted","Data":"b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6"} Sep 16 07:47:52.123861 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:52.123591 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:52.145453 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:52.145406 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" podStartSLOduration=1.8138215469999999 podStartE2EDuration="2.145392701s" totalImagesPullingTime="331.571154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:52.143232024 +0000 UTC m=+845.062847971" watchObservedRunningTime="2026-09-16 07:47:52.145392701 +0000 UTC m=+845.065008646" Sep 16 07:47:58.134935 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:58.134905 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:47:59.106984 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:47:59.106947 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:48:21.899720 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:21.899676 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:48:28.529261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.529225 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b"] Sep 16 07:48:28.529701 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.529451 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" podUID="e60025a6-82c1-494e-bb90-d7880e88fbe4" containerName="maas-api" containerID="cri-o://b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6" gracePeriod=30 Sep 16 07:48:28.791219 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.791150 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:48:28.893243 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.890434 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh85r\" (UniqueName: \"kubernetes.io/projected/e60025a6-82c1-494e-bb90-d7880e88fbe4-kube-api-access-xh85r\") pod \"e60025a6-82c1-494e-bb90-d7880e88fbe4\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " Sep 16 07:48:28.893243 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.891152 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-metrics-certs\") pod \"e60025a6-82c1-494e-bb90-d7880e88fbe4\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " Sep 16 07:48:28.893243 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.891317 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-maas-api-tls\") pod \"e60025a6-82c1-494e-bb90-d7880e88fbe4\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " Sep 16 07:48:28.893243 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.891407 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60025a6-82c1-494e-bb90-d7880e88fbe4-openshift-service-ca\") pod \"e60025a6-82c1-494e-bb90-d7880e88fbe4\" (UID: \"e60025a6-82c1-494e-bb90-d7880e88fbe4\") " Sep 16 07:48:28.893243 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.892054 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60025a6-82c1-494e-bb90-d7880e88fbe4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e60025a6-82c1-494e-bb90-d7880e88fbe4" (UID: "e60025a6-82c1-494e-bb90-d7880e88fbe4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:48:28.895618 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.895587 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e60025a6-82c1-494e-bb90-d7880e88fbe4" (UID: "e60025a6-82c1-494e-bb90-d7880e88fbe4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:48:28.897580 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.897543 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e60025a6-82c1-494e-bb90-d7880e88fbe4" (UID: "e60025a6-82c1-494e-bb90-d7880e88fbe4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:48:28.898677 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.898648 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60025a6-82c1-494e-bb90-d7880e88fbe4-kube-api-access-xh85r" (OuterVolumeSpecName: "kube-api-access-xh85r") pod "e60025a6-82c1-494e-bb90-d7880e88fbe4" (UID: "e60025a6-82c1-494e-bb90-d7880e88fbe4"). InnerVolumeSpecName "kube-api-access-xh85r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:48:28.992269 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.992205 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:48:28.992269 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.992268 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60025a6-82c1-494e-bb90-d7880e88fbe4-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:48:28.992512 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.992284 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh85r\" (UniqueName: \"kubernetes.io/projected/e60025a6-82c1-494e-bb90-d7880e88fbe4-kube-api-access-xh85r\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:48:28.992512 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:28.992306 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e60025a6-82c1-494e-bb90-d7880e88fbe4-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:48:29.348281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:29.348245 2580 generic.go:356] "Generic (PLEG): container finished" podID="e60025a6-82c1-494e-bb90-d7880e88fbe4" containerID="b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6" exitCode=0 Sep 16 07:48:29.348516 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:29.348295 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" Sep 16 07:48:29.348516 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:29.348332 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" event={"ID":"e60025a6-82c1-494e-bb90-d7880e88fbe4","Type":"ContainerDied","Data":"b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6"} Sep 16 07:48:29.348516 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:29.348388 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b" event={"ID":"e60025a6-82c1-494e-bb90-d7880e88fbe4","Type":"ContainerDied","Data":"ea56a9007d85776547659605cde3b85380410e6b382bd4709889f7d50df5fd58"} Sep 16 07:48:29.348516 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:29.348418 2580 scope.go:117] "RemoveContainer" containerID="b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6" Sep 16 07:48:29.358433 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:29.358415 2580 scope.go:117] "RemoveContainer" containerID="b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6" Sep 16 07:48:29.358695 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:48:29.358676 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6\": container with ID starting with b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6 not found: ID does not exist" containerID="b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6" Sep 16 07:48:29.358753 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:29.358707 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6"} err="failed to get container status \"b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6\": rpc error: code = NotFound desc = could not find container \"b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6\": container with ID starting with b2decb726cadb5d468da7b3b380ec8b717706dfd6b1c7444904a2255d11728a6 not found: ID does not exist" Sep 16 07:48:29.371980 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:29.371958 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b"] Sep 16 07:48:29.377432 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:29.377413 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5fb973f-d67c94995-9jv2b"] Sep 16 07:48:29.560528 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:29.560500 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e60025a6-82c1-494e-bb90-d7880e88fbe4" path="/var/lib/kubelet/pods/e60025a6-82c1-494e-bb90-d7880e88fbe4/volumes" Sep 16 07:48:34.196261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.196199 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw"] Sep 16 07:48:34.196653 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.196637 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60025a6-82c1-494e-bb90-d7880e88fbe4" containerName="maas-api" Sep 16 07:48:34.196653 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.196652 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60025a6-82c1-494e-bb90-d7880e88fbe4" containerName="maas-api" Sep 16 07:48:34.196743 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.196731 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60025a6-82c1-494e-bb90-d7880e88fbe4" containerName="maas-api" Sep 16 07:48:34.200265 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.200239 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.202775 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.202751 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4d9f69e4-serving-cert\"" Sep 16 07:48:34.215970 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.215948 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw"] Sep 16 07:48:34.243687 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.243662 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-maas-api-tls\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.243798 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.243713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-metrics-certs\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.243798 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.243739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrxq\" (UniqueName: \"kubernetes.io/projected/9817a027-32a6-4bb6-a93b-94b96daec891-kube-api-access-2vrxq\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.243798 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.243780 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817a027-32a6-4bb6-a93b-94b96daec891-openshift-service-ca\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.344376 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.344345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817a027-32a6-4bb6-a93b-94b96daec891-openshift-service-ca\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.344498 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.344450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-maas-api-tls\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.344564 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.344506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-metrics-certs\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.344620 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:48:34.344607 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-serving-cert: secret "maas-api-e2e-ait-4d9f69e4-serving-cert" not found Sep 16 07:48:34.344679 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.344622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrxq\" (UniqueName: \"kubernetes.io/projected/9817a027-32a6-4bb6-a93b-94b96daec891-kube-api-access-2vrxq\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.344679 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:48:34.344676 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-maas-api-tls podName:9817a027-32a6-4bb6-a93b-94b96daec891 nodeName:}" failed. No retries permitted until 2026-09-16 07:48:34.844655504 +0000 UTC m=+887.764271450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-maas-api-tls") pod "maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" (UID: "9817a027-32a6-4bb6-a93b-94b96daec891") : secret "maas-api-e2e-ait-4d9f69e4-serving-cert" not found Sep 16 07:48:34.345076 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.345054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817a027-32a6-4bb6-a93b-94b96daec891-openshift-service-ca\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.346802 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.346775 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-metrics-certs\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.355785 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.355765 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrxq\" (UniqueName: \"kubernetes.io/projected/9817a027-32a6-4bb6-a93b-94b96daec891-kube-api-access-2vrxq\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.849757 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.849726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-maas-api-tls\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:34.851982 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:34.851957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-maas-api-tls\") pod \"maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:35.138680 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:35.138598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:35.412719 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:35.412684 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw"] Sep 16 07:48:35.414247 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:48:35.414198 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9817a027_32a6_4bb6_a93b_94b96daec891.slice/crio-9c7c5057560425acfbe218b031636a2a666a73a70b33fa0b4d66177643d5e124 WatchSource:0}: Error finding container 9c7c5057560425acfbe218b031636a2a666a73a70b33fa0b4d66177643d5e124: Status 404 returned error can't find the container with id 9c7c5057560425acfbe218b031636a2a666a73a70b33fa0b4d66177643d5e124 Sep 16 07:48:36.394007 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:36.393966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" event={"ID":"9817a027-32a6-4bb6-a93b-94b96daec891","Type":"ContainerStarted","Data":"f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a"} Sep 16 07:48:36.394173 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:36.394047 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" event={"ID":"9817a027-32a6-4bb6-a93b-94b96daec891","Type":"ContainerStarted","Data":"9c7c5057560425acfbe218b031636a2a666a73a70b33fa0b4d66177643d5e124"} Sep 16 07:48:36.394173 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:36.394078 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:36.413739 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:36.413693 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" podStartSLOduration=2.031482328 podStartE2EDuration="2.413678447s" totalImagesPullingTime="382.196119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:48:36.410956524 +0000 UTC m=+889.330572482" watchObservedRunningTime="2026-09-16 07:48:36.413678447 +0000 UTC m=+889.333294434" Sep 16 07:48:36.801550 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:36.801512 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:48:42.405920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:42.405839 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:48:47.504772 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:47.504739 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:48:47.506031 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:47.506008 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:48:52.609857 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:48:52.609821 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:49:00.868366 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.868330 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn"] Sep 16 07:49:00.873147 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.873124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:00.878360 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.877997 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5dd30f\"/\"default-dockercfg-q6svq\"" Sep 16 07:49:00.878360 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.878010 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5dd30f\"/\"kube-root-ca.crt\"" Sep 16 07:49:00.878360 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.878134 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5dd30f\"/\"openshift-service-ca.crt\"" Sep 16 07:49:00.878360 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.878315 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5dd30f\"/\"test-model-5dd30f-kserve-self-signed-certs\"" Sep 16 07:49:00.882190 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.881790 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn"] Sep 16 07:49:00.988590 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.988558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-dshm\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:00.988759 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.988620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-home\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:00.988759 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.988678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-tmp-dir\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:00.988759 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.988699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b02c97c-ca40-4aed-9814-54e40eceadd2-tls-certs\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:00.988881 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.988767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-model-cache\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:00.988881 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:00.988810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhwp4\" (UniqueName: \"kubernetes.io/projected/1b02c97c-ca40-4aed-9814-54e40eceadd2-kube-api-access-nhwp4\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.089644 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.089611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-dshm\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.089808 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.089668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-home\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.089865 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.089843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-tmp-dir\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.089903 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.089887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b02c97c-ca40-4aed-9814-54e40eceadd2-tls-certs\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.089955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.089915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-model-cache\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.089995 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.089956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwp4\" (UniqueName: \"kubernetes.io/projected/1b02c97c-ca40-4aed-9814-54e40eceadd2-kube-api-access-nhwp4\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.090118 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.090089 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-home\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.090275 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.090176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-tmp-dir\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.090334 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.090310 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-model-cache\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.092313 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.092288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-dshm\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.092588 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.092567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b02c97c-ca40-4aed-9814-54e40eceadd2-tls-certs\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.100017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.099993 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhwp4\" (UniqueName: \"kubernetes.io/projected/1b02c97c-ca40-4aed-9814-54e40eceadd2-kube-api-access-nhwp4\") pod \"test-model-5dd30f-kserve-75987f849c-mncsn\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.184882 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.184851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:01.461933 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.460553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn"] Sep 16 07:49:01.464323 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:49:01.464289 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b02c97c_ca40_4aed_9814_54e40eceadd2.slice/crio-858810ab777ffa6eea47922178ea5df3c1cf0c86e54c488e7edef9594775bde1 WatchSource:0}: Error finding container 858810ab777ffa6eea47922178ea5df3c1cf0c86e54c488e7edef9594775bde1: Status 404 returned error can't find the container with id 858810ab777ffa6eea47922178ea5df3c1cf0c86e54c488e7edef9594775bde1 Sep 16 07:49:01.547529 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:01.547492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" event={"ID":"1b02c97c-ca40-4aed-9814-54e40eceadd2","Type":"ContainerStarted","Data":"858810ab777ffa6eea47922178ea5df3c1cf0c86e54c488e7edef9594775bde1"} Sep 16 07:49:02.557027 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:02.557001 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" event={"ID":"1b02c97c-ca40-4aed-9814-54e40eceadd2","Type":"ContainerStarted","Data":"1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87"} Sep 16 07:49:02.578264 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:02.578197 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" podStartSLOduration=1.604916426 podStartE2EDuration="2.578182086s" totalImagesPullingTime="973.26566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:49:02.574714545 +0000 UTC m=+915.494330492" watchObservedRunningTime="2026-09-16 07:49:02.578182086 +0000 UTC m=+915.497798031" Sep 16 07:49:04.150551 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.150516 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw"] Sep 16 07:49:04.151059 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.150760 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" podUID="9817a027-32a6-4bb6-a93b-94b96daec891" containerName="maas-api" containerID="cri-o://f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a" gracePeriod=30 Sep 16 07:49:04.414154 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.414128 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:49:04.421541 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.421517 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vrxq\" (UniqueName: \"kubernetes.io/projected/9817a027-32a6-4bb6-a93b-94b96daec891-kube-api-access-2vrxq\") pod \"9817a027-32a6-4bb6-a93b-94b96daec891\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " Sep 16 07:49:04.421673 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.421560 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-metrics-certs\") pod \"9817a027-32a6-4bb6-a93b-94b96daec891\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " Sep 16 07:49:04.421673 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.421584 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-maas-api-tls\") pod \"9817a027-32a6-4bb6-a93b-94b96daec891\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " Sep 16 07:49:04.421673 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.421626 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817a027-32a6-4bb6-a93b-94b96daec891-openshift-service-ca\") pod \"9817a027-32a6-4bb6-a93b-94b96daec891\" (UID: \"9817a027-32a6-4bb6-a93b-94b96daec891\") " Sep 16 07:49:04.422016 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.421991 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9817a027-32a6-4bb6-a93b-94b96daec891-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9817a027-32a6-4bb6-a93b-94b96daec891" (UID: "9817a027-32a6-4bb6-a93b-94b96daec891"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:49:04.423660 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.423635 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9817a027-32a6-4bb6-a93b-94b96daec891-kube-api-access-2vrxq" (OuterVolumeSpecName: "kube-api-access-2vrxq") pod "9817a027-32a6-4bb6-a93b-94b96daec891" (UID: "9817a027-32a6-4bb6-a93b-94b96daec891"). InnerVolumeSpecName "kube-api-access-2vrxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:49:04.423838 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.423817 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9817a027-32a6-4bb6-a93b-94b96daec891" (UID: "9817a027-32a6-4bb6-a93b-94b96daec891"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:49:04.423922 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.423841 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9817a027-32a6-4bb6-a93b-94b96daec891" (UID: "9817a027-32a6-4bb6-a93b-94b96daec891"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:49:04.522639 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.522607 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vrxq\" (UniqueName: \"kubernetes.io/projected/9817a027-32a6-4bb6-a93b-94b96daec891-kube-api-access-2vrxq\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:04.522639 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.522638 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:04.522844 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.522648 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9817a027-32a6-4bb6-a93b-94b96daec891-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:04.522844 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.522656 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9817a027-32a6-4bb6-a93b-94b96daec891-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:04.571434 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.571404 2580 generic.go:356] "Generic (PLEG): container finished" podID="9817a027-32a6-4bb6-a93b-94b96daec891" containerID="f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a" exitCode=0 Sep 16 07:49:04.571609 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.571445 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" Sep 16 07:49:04.571609 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.571487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" event={"ID":"9817a027-32a6-4bb6-a93b-94b96daec891","Type":"ContainerDied","Data":"f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a"} Sep 16 07:49:04.571609 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.571536 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw" event={"ID":"9817a027-32a6-4bb6-a93b-94b96daec891","Type":"ContainerDied","Data":"9c7c5057560425acfbe218b031636a2a666a73a70b33fa0b4d66177643d5e124"} Sep 16 07:49:04.571609 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.571560 2580 scope.go:117] "RemoveContainer" containerID="f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a" Sep 16 07:49:04.581300 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.581281 2580 scope.go:117] "RemoveContainer" containerID="f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a" Sep 16 07:49:04.581595 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:49:04.581574 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a\": container with ID starting with f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a not found: ID does not exist" containerID="f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a" Sep 16 07:49:04.581676 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.581602 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a"} err="failed to get container status \"f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a\": rpc error: code = NotFound desc = could not find container \"f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a\": container with ID starting with f05e4c250e9e22cd2760f9316843ad9fe16c7b06224a4bf79b10f5ceebc09f6a not found: ID does not exist" Sep 16 07:49:04.595373 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.595346 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw"] Sep 16 07:49:04.600951 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:04.600924 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9f69e4-b7cc88f6d-mz6dw"] Sep 16 07:49:05.561523 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:05.561483 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9817a027-32a6-4bb6-a93b-94b96daec891" path="/var/lib/kubelet/pods/9817a027-32a6-4bb6-a93b-94b96daec891/volumes" Sep 16 07:49:06.833919 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:06.833866 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:49:10.843321 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.843290 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v"] Sep 16 07:49:10.843804 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.843794 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9817a027-32a6-4bb6-a93b-94b96daec891" containerName="maas-api" Sep 16 07:49:10.843877 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.843808 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9817a027-32a6-4bb6-a93b-94b96daec891" containerName="maas-api" Sep 16 07:49:10.843942 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.843895 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9817a027-32a6-4bb6-a93b-94b96daec891" containerName="maas-api" Sep 16 07:49:10.848661 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.848639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.851129 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.851058 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1892c35b-serving-cert\"" Sep 16 07:49:10.855899 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.855869 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v"] Sep 16 07:49:10.876203 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.876177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171b7ace-e746-4f8d-82db-9307ff44af67-openshift-service-ca\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.876361 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.876235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-maas-api-tls\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.876433 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.876360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xlk\" (UniqueName: \"kubernetes.io/projected/171b7ace-e746-4f8d-82db-9307ff44af67-kube-api-access-t7xlk\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.876433 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.876411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-metrics-certs\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.977159 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.977131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-maas-api-tls\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.977355 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.977195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xlk\" (UniqueName: \"kubernetes.io/projected/171b7ace-e746-4f8d-82db-9307ff44af67-kube-api-access-t7xlk\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.977355 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.977241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-metrics-certs\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.977466 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.977383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171b7ace-e746-4f8d-82db-9307ff44af67-openshift-service-ca\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.978049 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.978025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171b7ace-e746-4f8d-82db-9307ff44af67-openshift-service-ca\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.979603 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.979583 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-metrics-certs\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.979709 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.979693 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-maas-api-tls\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:10.984789 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:10.984772 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xlk\" (UniqueName: \"kubernetes.io/projected/171b7ace-e746-4f8d-82db-9307ff44af67-kube-api-access-t7xlk\") pod \"maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:11.163060 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:11.163023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:11.185518 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:11.185486 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:11.185628 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:11.185535 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:11.198573 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:11.198549 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:11.439741 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:11.439712 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v"] Sep 16 07:49:11.441099 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:49:11.441073 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171b7ace_e746_4f8d_82db_9307ff44af67.slice/crio-789d672e9c4e0b37242275c35b0a46ff2eb022de678651de981d8363ac1dd9fd WatchSource:0}: Error finding container 789d672e9c4e0b37242275c35b0a46ff2eb022de678651de981d8363ac1dd9fd: Status 404 returned error can't find the container with id 789d672e9c4e0b37242275c35b0a46ff2eb022de678651de981d8363ac1dd9fd Sep 16 07:49:11.621199 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:11.621159 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" event={"ID":"171b7ace-e746-4f8d-82db-9307ff44af67","Type":"ContainerStarted","Data":"789d672e9c4e0b37242275c35b0a46ff2eb022de678651de981d8363ac1dd9fd"} Sep 16 07:49:11.633287 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:11.633262 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:49:12.629638 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:12.629602 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" event={"ID":"171b7ace-e746-4f8d-82db-9307ff44af67","Type":"ContainerStarted","Data":"9459828fe7214d060a3f3fcd7f1147e2440f0240466f84853965d70f38420d92"} Sep 16 07:49:12.648760 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:12.648712 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" podStartSLOduration=2.2363431990000002 podStartE2EDuration="2.648697191s" totalImagesPullingTime="412.353992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:49:12.646474191 +0000 UTC m=+925.566090147" watchObservedRunningTime="2026-09-16 07:49:12.648697191 +0000 UTC m=+925.568313137" Sep 16 07:49:13.635187 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:13.635159 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:19.005170 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.005136 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb"] Sep 16 07:49:19.005574 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.005408 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" podUID="5f876c73-a707-4772-9dcb-b6bcf412cc90" containerName="maas-api" containerID="cri-o://a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6" gracePeriod=30 Sep 16 07:49:19.272035 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.271971 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:49:19.348271 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.348236 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-metrics-certs\") pod \"5f876c73-a707-4772-9dcb-b6bcf412cc90\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " Sep 16 07:49:19.348438 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.348295 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-maas-api-tls\") pod \"5f876c73-a707-4772-9dcb-b6bcf412cc90\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " Sep 16 07:49:19.348438 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.348364 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhvcv\" (UniqueName: \"kubernetes.io/projected/5f876c73-a707-4772-9dcb-b6bcf412cc90-kube-api-access-jhvcv\") pod \"5f876c73-a707-4772-9dcb-b6bcf412cc90\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " Sep 16 07:49:19.348438 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.348433 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f876c73-a707-4772-9dcb-b6bcf412cc90-openshift-service-ca\") pod \"5f876c73-a707-4772-9dcb-b6bcf412cc90\" (UID: \"5f876c73-a707-4772-9dcb-b6bcf412cc90\") " Sep 16 07:49:19.348819 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.348795 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f876c73-a707-4772-9dcb-b6bcf412cc90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f876c73-a707-4772-9dcb-b6bcf412cc90" (UID: "5f876c73-a707-4772-9dcb-b6bcf412cc90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:49:19.350400 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.350369 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5f876c73-a707-4772-9dcb-b6bcf412cc90" (UID: "5f876c73-a707-4772-9dcb-b6bcf412cc90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:49:19.350489 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.350413 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f876c73-a707-4772-9dcb-b6bcf412cc90-kube-api-access-jhvcv" (OuterVolumeSpecName: "kube-api-access-jhvcv") pod "5f876c73-a707-4772-9dcb-b6bcf412cc90" (UID: "5f876c73-a707-4772-9dcb-b6bcf412cc90"). InnerVolumeSpecName "kube-api-access-jhvcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:49:19.350489 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.350418 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f876c73-a707-4772-9dcb-b6bcf412cc90" (UID: "5f876c73-a707-4772-9dcb-b6bcf412cc90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:49:19.449509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.449480 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f876c73-a707-4772-9dcb-b6bcf412cc90-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:19.449509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.449510 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:19.451421 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.449521 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f876c73-a707-4772-9dcb-b6bcf412cc90-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:19.451421 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.449529 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhvcv\" (UniqueName: \"kubernetes.io/projected/5f876c73-a707-4772-9dcb-b6bcf412cc90-kube-api-access-jhvcv\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:19.647306 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.647281 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:19.672806 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.672777 2580 generic.go:356] "Generic (PLEG): container finished" podID="5f876c73-a707-4772-9dcb-b6bcf412cc90" containerID="a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6" exitCode=0 Sep 16 07:49:19.672942 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.672820 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" Sep 16 07:49:19.672942 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.672839 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" event={"ID":"5f876c73-a707-4772-9dcb-b6bcf412cc90","Type":"ContainerDied","Data":"a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6"} Sep 16 07:49:19.672942 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.672877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb" event={"ID":"5f876c73-a707-4772-9dcb-b6bcf412cc90","Type":"ContainerDied","Data":"c9b61a5f87a28d74e9a7f0f90824ebbf95e5c3199a9faea6abe2976a6de2c9df"} Sep 16 07:49:19.672942 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.672898 2580 scope.go:117] "RemoveContainer" containerID="a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6" Sep 16 07:49:19.683512 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.683486 2580 scope.go:117] "RemoveContainer" containerID="a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6" Sep 16 07:49:19.683860 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:49:19.683833 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6\": container with ID starting with a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6 not found: ID does not exist" containerID="a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6" Sep 16 07:49:19.683974 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.683868 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6"} err="failed to get container status \"a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6\": rpc error: code = NotFound desc = could not find container \"a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6\": container with ID starting with a349d90c0f57fec8f381d512dac6e24cf069a19228e31794f1ae6faa1110f4e6 not found: ID does not exist" Sep 16 07:49:19.697153 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.697106 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb"] Sep 16 07:49:19.703610 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:19.703586 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b60434-5c9f97dc45-rjttb"] Sep 16 07:49:21.560742 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:21.560705 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f876c73-a707-4772-9dcb-b6bcf412cc90" path="/var/lib/kubelet/pods/5f876c73-a707-4772-9dcb-b6bcf412cc90/volumes" Sep 16 07:49:30.910351 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:30.910319 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:49:34.539302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.539259 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v"] Sep 16 07:49:34.540161 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.539560 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" podUID="171b7ace-e746-4f8d-82db-9307ff44af67" containerName="maas-api" containerID="cri-o://9459828fe7214d060a3f3fcd7f1147e2440f0240466f84853965d70f38420d92" gracePeriod=30 Sep 16 07:49:34.641607 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.641561 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" podUID="171b7ace-e746-4f8d-82db-9307ff44af67" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.71:8443/health\": dial tcp 10.132.0.71:8443: connect: connection refused" Sep 16 07:49:34.768520 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.768480 2580 generic.go:356] "Generic (PLEG): container finished" podID="171b7ace-e746-4f8d-82db-9307ff44af67" containerID="9459828fe7214d060a3f3fcd7f1147e2440f0240466f84853965d70f38420d92" exitCode=0 Sep 16 07:49:34.768686 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.768567 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" event={"ID":"171b7ace-e746-4f8d-82db-9307ff44af67","Type":"ContainerDied","Data":"9459828fe7214d060a3f3fcd7f1147e2440f0240466f84853965d70f38420d92"} Sep 16 07:49:34.796375 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.796323 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:34.883575 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.883543 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-metrics-certs\") pod \"171b7ace-e746-4f8d-82db-9307ff44af67\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " Sep 16 07:49:34.883755 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.883606 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7xlk\" (UniqueName: \"kubernetes.io/projected/171b7ace-e746-4f8d-82db-9307ff44af67-kube-api-access-t7xlk\") pod \"171b7ace-e746-4f8d-82db-9307ff44af67\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " Sep 16 07:49:34.883755 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.883739 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171b7ace-e746-4f8d-82db-9307ff44af67-openshift-service-ca\") pod \"171b7ace-e746-4f8d-82db-9307ff44af67\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " Sep 16 07:49:34.883895 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.883879 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-maas-api-tls\") pod \"171b7ace-e746-4f8d-82db-9307ff44af67\" (UID: \"171b7ace-e746-4f8d-82db-9307ff44af67\") " Sep 16 07:49:34.884318 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.884289 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171b7ace-e746-4f8d-82db-9307ff44af67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "171b7ace-e746-4f8d-82db-9307ff44af67" (UID: "171b7ace-e746-4f8d-82db-9307ff44af67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:49:34.885849 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.885819 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "171b7ace-e746-4f8d-82db-9307ff44af67" (UID: "171b7ace-e746-4f8d-82db-9307ff44af67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:49:34.887318 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.887291 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171b7ace-e746-4f8d-82db-9307ff44af67-kube-api-access-t7xlk" (OuterVolumeSpecName: "kube-api-access-t7xlk") pod "171b7ace-e746-4f8d-82db-9307ff44af67" (UID: "171b7ace-e746-4f8d-82db-9307ff44af67"). InnerVolumeSpecName "kube-api-access-t7xlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:49:34.887442 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.887406 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "171b7ace-e746-4f8d-82db-9307ff44af67" (UID: "171b7ace-e746-4f8d-82db-9307ff44af67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:49:34.984774 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.984745 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:34.984774 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.984769 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171b7ace-e746-4f8d-82db-9307ff44af67-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:34.984774 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.984779 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xlk\" (UniqueName: \"kubernetes.io/projected/171b7ace-e746-4f8d-82db-9307ff44af67-kube-api-access-t7xlk\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:34.985062 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:34.984790 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171b7ace-e746-4f8d-82db-9307ff44af67-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:49:35.776300 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:35.776269 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" Sep 16 07:49:35.776300 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:35.776284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v" event={"ID":"171b7ace-e746-4f8d-82db-9307ff44af67","Type":"ContainerDied","Data":"789d672e9c4e0b37242275c35b0a46ff2eb022de678651de981d8363ac1dd9fd"} Sep 16 07:49:35.776715 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:35.776346 2580 scope.go:117] "RemoveContainer" containerID="9459828fe7214d060a3f3fcd7f1147e2440f0240466f84853965d70f38420d92" Sep 16 07:49:35.809978 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:35.809951 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v"] Sep 16 07:49:35.819740 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:35.819690 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1892c35b-74dc99fd5c-gxf6v"] Sep 16 07:49:37.561739 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:37.561708 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="171b7ace-e746-4f8d-82db-9307ff44af67" path="/var/lib/kubelet/pods/171b7ace-e746-4f8d-82db-9307ff44af67/volumes" Sep 16 07:49:50.922007 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:49:50.921968 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:50:08.544963 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.544931 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz"] Sep 16 07:50:08.545438 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.545402 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171b7ace-e746-4f8d-82db-9307ff44af67" containerName="maas-api" Sep 16 07:50:08.545438 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.545415 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="171b7ace-e746-4f8d-82db-9307ff44af67" containerName="maas-api" Sep 16 07:50:08.545511 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.545441 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f876c73-a707-4772-9dcb-b6bcf412cc90" containerName="maas-api" Sep 16 07:50:08.545511 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.545448 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f876c73-a707-4772-9dcb-b6bcf412cc90" containerName="maas-api" Sep 16 07:50:08.545580 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.545523 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="171b7ace-e746-4f8d-82db-9307ff44af67" containerName="maas-api" Sep 16 07:50:08.545580 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.545537 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f876c73-a707-4772-9dcb-b6bcf412cc90" containerName="maas-api" Sep 16 07:50:08.550153 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.550127 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.553265 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.553240 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-82bbbf-serving-cert\"" Sep 16 07:50:08.559762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.559737 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz"] Sep 16 07:50:08.569778 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.569755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083dde50-db57-461b-bd17-74e8f3335b10-openshift-service-ca\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.569882 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.569787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4plz\" (UniqueName: \"kubernetes.io/projected/083dde50-db57-461b-bd17-74e8f3335b10-kube-api-access-d4plz\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.569882 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.569821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-metrics-certs\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.569993 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.569959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-maas-api-tls\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.670845 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.670813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083dde50-db57-461b-bd17-74e8f3335b10-openshift-service-ca\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.671019 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.670848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4plz\" (UniqueName: \"kubernetes.io/projected/083dde50-db57-461b-bd17-74e8f3335b10-kube-api-access-d4plz\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.671019 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.670899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-metrics-certs\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.671019 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.670965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-maas-api-tls\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.671240 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:50:08.671090 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-serving-cert: secret "maas-api-e2e-explicit-82bbbf-serving-cert" not found Sep 16 07:50:08.671240 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:50:08.671155 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-maas-api-tls podName:083dde50-db57-461b-bd17-74e8f3335b10 nodeName:}" failed. No retries permitted until 2026-09-16 07:50:09.171134375 +0000 UTC m=+982.090750302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-maas-api-tls") pod "maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" (UID: "083dde50-db57-461b-bd17-74e8f3335b10") : secret "maas-api-e2e-explicit-82bbbf-serving-cert" not found Sep 16 07:50:08.671585 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.671561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083dde50-db57-461b-bd17-74e8f3335b10-openshift-service-ca\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.673306 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.673287 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-metrics-certs\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:08.679434 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:08.679417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4plz\" (UniqueName: \"kubernetes.io/projected/083dde50-db57-461b-bd17-74e8f3335b10-kube-api-access-d4plz\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:09.176334 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:09.176298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-maas-api-tls\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:09.178739 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:09.178715 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-maas-api-tls\") pod \"maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:09.462802 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:09.462720 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:09.734757 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:09.734726 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz"] Sep 16 07:50:09.736075 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:50:09.736049 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083dde50_db57_461b_bd17_74e8f3335b10.slice/crio-23e83f4665cf15a924f2378a4cf35ea8e1e45a91eda82d5aa48f4107e37ddb3a WatchSource:0}: Error finding container 23e83f4665cf15a924f2378a4cf35ea8e1e45a91eda82d5aa48f4107e37ddb3a: Status 404 returned error can't find the container with id 23e83f4665cf15a924f2378a4cf35ea8e1e45a91eda82d5aa48f4107e37ddb3a Sep 16 07:50:09.737674 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:09.737652 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:50:09.986051 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:09.985946 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" event={"ID":"083dde50-db57-461b-bd17-74e8f3335b10","Type":"ContainerStarted","Data":"23e83f4665cf15a924f2378a4cf35ea8e1e45a91eda82d5aa48f4107e37ddb3a"} Sep 16 07:50:10.994317 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:10.994279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" event={"ID":"083dde50-db57-461b-bd17-74e8f3335b10","Type":"ContainerStarted","Data":"1a1cfa36af80f29c8238cce8bc31f3e4b4332c5ab90b3226c94d9abd0919fd93"} Sep 16 07:50:10.994689 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:10.994368 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:11.015392 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:11.015344 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" podStartSLOduration=2.534517461 podStartE2EDuration="3.015329573s" totalImagesPullingTime="480.812112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:50:11.011930667 +0000 UTC m=+983.931546618" watchObservedRunningTime="2026-09-16 07:50:11.015329573 +0000 UTC m=+983.934945519" Sep 16 07:50:17.006639 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:17.006607 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:50:17.608058 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:17.608022 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:50:37.315752 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:50:37.315716 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:51:01.305864 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:01.305822 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:51:20.450292 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.450254 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x"] Sep 16 07:51:20.454050 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.454026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.456584 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.456566 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b71faf9e-serving-cert\"" Sep 16 07:51:20.466925 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.466898 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x"] Sep 16 07:51:20.612751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.612719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396304ef-8e18-48df-b0ab-d78b8c4afd46-openshift-service-ca\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.612925 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.612754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-metrics-certs\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.612925 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.612802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wwm4\" (UniqueName: \"kubernetes.io/projected/396304ef-8e18-48df-b0ab-d78b8c4afd46-kube-api-access-6wwm4\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.613023 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.612919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-maas-api-tls\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.713631 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.713559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-maas-api-tls\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.713631 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.713608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396304ef-8e18-48df-b0ab-d78b8c4afd46-openshift-service-ca\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.713631 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.713624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-metrics-certs\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.713884 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.713665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wwm4\" (UniqueName: \"kubernetes.io/projected/396304ef-8e18-48df-b0ab-d78b8c4afd46-kube-api-access-6wwm4\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.714326 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.714304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396304ef-8e18-48df-b0ab-d78b8c4afd46-openshift-service-ca\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.715997 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.715969 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-maas-api-tls\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.716148 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.716125 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-metrics-certs\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.722043 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.722022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wwm4\" (UniqueName: \"kubernetes.io/projected/396304ef-8e18-48df-b0ab-d78b8c4afd46-kube-api-access-6wwm4\") pod \"maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:20.769896 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:20.769871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:21.045477 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:21.045445 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x"] Sep 16 07:51:21.049247 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:51:21.046750 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396304ef_8e18_48df_b0ab_d78b8c4afd46.slice/crio-5df92fe588d15f9deb8cb19df1c3cacd01a1945b6b021580708f09c29ad96169 WatchSource:0}: Error finding container 5df92fe588d15f9deb8cb19df1c3cacd01a1945b6b021580708f09c29ad96169: Status 404 returned error can't find the container with id 5df92fe588d15f9deb8cb19df1c3cacd01a1945b6b021580708f09c29ad96169 Sep 16 07:51:21.422717 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:21.422681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" event={"ID":"396304ef-8e18-48df-b0ab-d78b8c4afd46","Type":"ContainerStarted","Data":"5df92fe588d15f9deb8cb19df1c3cacd01a1945b6b021580708f09c29ad96169"} Sep 16 07:51:22.429552 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:22.429514 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" event={"ID":"396304ef-8e18-48df-b0ab-d78b8c4afd46","Type":"ContainerStarted","Data":"4862743efb21153f643c517dfb689760429f663248118261598a0281fdb67c5e"} Sep 16 07:51:22.430004 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:22.429587 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:22.449785 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:22.449736 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" podStartSLOduration=1.9613990810000002 podStartE2EDuration="2.44971933s" totalImagesPullingTime="488.320249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:51:22.447256676 +0000 UTC m=+1055.366872623" watchObservedRunningTime="2026-09-16 07:51:22.44971933 +0000 UTC m=+1055.369335275" Sep 16 07:51:28.441254 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:28.441227 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:51:31.815770 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:51:31.815731 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:52:01.211995 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:01.211962 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:52:04.504756 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.504722 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x"] Sep 16 07:52:04.505243 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.504982 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" podUID="396304ef-8e18-48df-b0ab-d78b8c4afd46" containerName="maas-api" containerID="cri-o://4862743efb21153f643c517dfb689760429f663248118261598a0281fdb67c5e" gracePeriod=30 Sep 16 07:52:04.688854 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.688813 2580 generic.go:356] "Generic (PLEG): container finished" podID="396304ef-8e18-48df-b0ab-d78b8c4afd46" containerID="4862743efb21153f643c517dfb689760429f663248118261598a0281fdb67c5e" exitCode=0 Sep 16 07:52:04.689023 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.688932 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" event={"ID":"396304ef-8e18-48df-b0ab-d78b8c4afd46","Type":"ContainerDied","Data":"4862743efb21153f643c517dfb689760429f663248118261598a0281fdb67c5e"} Sep 16 07:52:04.774861 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.774831 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:52:04.908345 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.908306 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396304ef-8e18-48df-b0ab-d78b8c4afd46-openshift-service-ca\") pod \"396304ef-8e18-48df-b0ab-d78b8c4afd46\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " Sep 16 07:52:04.908522 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.908354 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-maas-api-tls\") pod \"396304ef-8e18-48df-b0ab-d78b8c4afd46\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " Sep 16 07:52:04.908522 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.908395 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wwm4\" (UniqueName: \"kubernetes.io/projected/396304ef-8e18-48df-b0ab-d78b8c4afd46-kube-api-access-6wwm4\") pod \"396304ef-8e18-48df-b0ab-d78b8c4afd46\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " Sep 16 07:52:04.908522 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.908512 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-metrics-certs\") pod \"396304ef-8e18-48df-b0ab-d78b8c4afd46\" (UID: \"396304ef-8e18-48df-b0ab-d78b8c4afd46\") " Sep 16 07:52:04.908676 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.908621 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396304ef-8e18-48df-b0ab-d78b8c4afd46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "396304ef-8e18-48df-b0ab-d78b8c4afd46" (UID: "396304ef-8e18-48df-b0ab-d78b8c4afd46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:52:04.908859 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.908838 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396304ef-8e18-48df-b0ab-d78b8c4afd46-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:52:04.910489 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.910465 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "396304ef-8e18-48df-b0ab-d78b8c4afd46" (UID: "396304ef-8e18-48df-b0ab-d78b8c4afd46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:52:04.910663 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.910640 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "396304ef-8e18-48df-b0ab-d78b8c4afd46" (UID: "396304ef-8e18-48df-b0ab-d78b8c4afd46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:52:04.910747 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:04.910662 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396304ef-8e18-48df-b0ab-d78b8c4afd46-kube-api-access-6wwm4" (OuterVolumeSpecName: "kube-api-access-6wwm4") pod "396304ef-8e18-48df-b0ab-d78b8c4afd46" (UID: "396304ef-8e18-48df-b0ab-d78b8c4afd46"). InnerVolumeSpecName "kube-api-access-6wwm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:52:05.009864 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:05.009789 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wwm4\" (UniqueName: \"kubernetes.io/projected/396304ef-8e18-48df-b0ab-d78b8c4afd46-kube-api-access-6wwm4\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:52:05.009864 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:05.009815 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:52:05.009864 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:05.009824 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396304ef-8e18-48df-b0ab-d78b8c4afd46-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:52:05.696729 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:05.696694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" event={"ID":"396304ef-8e18-48df-b0ab-d78b8c4afd46","Type":"ContainerDied","Data":"5df92fe588d15f9deb8cb19df1c3cacd01a1945b6b021580708f09c29ad96169"} Sep 16 07:52:05.697196 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:05.696757 2580 scope.go:117] "RemoveContainer" containerID="4862743efb21153f643c517dfb689760429f663248118261598a0281fdb67c5e" Sep 16 07:52:05.697196 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:05.696707 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x" Sep 16 07:52:05.716141 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:05.716110 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x"] Sep 16 07:52:05.719709 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:05.719685 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b71faf9e-5657b675cc-lnt9x"] Sep 16 07:52:07.562099 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:07.562067 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="396304ef-8e18-48df-b0ab-d78b8c4afd46" path="/var/lib/kubelet/pods/396304ef-8e18-48df-b0ab-d78b8c4afd46/volumes" Sep 16 07:52:23.583997 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.583596 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz"] Sep 16 07:52:23.584820 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.584750 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" podUID="083dde50-db57-461b-bd17-74e8f3335b10" containerName="maas-api" containerID="cri-o://1a1cfa36af80f29c8238cce8bc31f3e4b4332c5ab90b3226c94d9abd0919fd93" gracePeriod=30 Sep 16 07:52:23.806322 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.806286 2580 generic.go:356] "Generic (PLEG): container finished" podID="083dde50-db57-461b-bd17-74e8f3335b10" containerID="1a1cfa36af80f29c8238cce8bc31f3e4b4332c5ab90b3226c94d9abd0919fd93" exitCode=0 Sep 16 07:52:23.806470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.806356 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" event={"ID":"083dde50-db57-461b-bd17-74e8f3335b10","Type":"ContainerDied","Data":"1a1cfa36af80f29c8238cce8bc31f3e4b4332c5ab90b3226c94d9abd0919fd93"} Sep 16 07:52:23.860518 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.860498 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:52:23.884295 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.883736 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-metrics-certs\") pod \"083dde50-db57-461b-bd17-74e8f3335b10\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " Sep 16 07:52:23.884295 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.883789 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4plz\" (UniqueName: \"kubernetes.io/projected/083dde50-db57-461b-bd17-74e8f3335b10-kube-api-access-d4plz\") pod \"083dde50-db57-461b-bd17-74e8f3335b10\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " Sep 16 07:52:23.884295 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.883838 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083dde50-db57-461b-bd17-74e8f3335b10-openshift-service-ca\") pod \"083dde50-db57-461b-bd17-74e8f3335b10\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " Sep 16 07:52:23.884295 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.883870 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-maas-api-tls\") pod \"083dde50-db57-461b-bd17-74e8f3335b10\" (UID: \"083dde50-db57-461b-bd17-74e8f3335b10\") " Sep 16 07:52:23.884796 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.884738 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083dde50-db57-461b-bd17-74e8f3335b10-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "083dde50-db57-461b-bd17-74e8f3335b10" (UID: "083dde50-db57-461b-bd17-74e8f3335b10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:52:23.886610 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.886583 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "083dde50-db57-461b-bd17-74e8f3335b10" (UID: "083dde50-db57-461b-bd17-74e8f3335b10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:52:23.886866 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.886834 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083dde50-db57-461b-bd17-74e8f3335b10-kube-api-access-d4plz" (OuterVolumeSpecName: "kube-api-access-d4plz") pod "083dde50-db57-461b-bd17-74e8f3335b10" (UID: "083dde50-db57-461b-bd17-74e8f3335b10"). InnerVolumeSpecName "kube-api-access-d4plz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:52:23.887339 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.887311 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "083dde50-db57-461b-bd17-74e8f3335b10" (UID: "083dde50-db57-461b-bd17-74e8f3335b10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:52:23.985289 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.985247 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083dde50-db57-461b-bd17-74e8f3335b10-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:52:23.985289 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.985290 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:52:23.985493 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.985305 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083dde50-db57-461b-bd17-74e8f3335b10-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:52:23.985493 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:23.985319 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4plz\" (UniqueName: \"kubernetes.io/projected/083dde50-db57-461b-bd17-74e8f3335b10-kube-api-access-d4plz\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:52:24.814185 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:24.814158 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" Sep 16 07:52:24.814185 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:24.814180 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz" event={"ID":"083dde50-db57-461b-bd17-74e8f3335b10","Type":"ContainerDied","Data":"23e83f4665cf15a924f2378a4cf35ea8e1e45a91eda82d5aa48f4107e37ddb3a"} Sep 16 07:52:24.814749 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:24.814294 2580 scope.go:117] "RemoveContainer" containerID="1a1cfa36af80f29c8238cce8bc31f3e4b4332c5ab90b3226c94d9abd0919fd93" Sep 16 07:52:24.837007 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:24.836978 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz"] Sep 16 07:52:24.841017 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:24.840997 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82bbbf-675d8c87fc-pbtwz"] Sep 16 07:52:25.561468 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:25.561432 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083dde50-db57-461b-bd17-74e8f3335b10" path="/var/lib/kubelet/pods/083dde50-db57-461b-bd17-74e8f3335b10/volumes" Sep 16 07:52:32.457889 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.457860 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr"] Sep 16 07:52:32.458339 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.458324 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083dde50-db57-461b-bd17-74e8f3335b10" containerName="maas-api" Sep 16 07:52:32.458402 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.458340 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="083dde50-db57-461b-bd17-74e8f3335b10" containerName="maas-api" Sep 16 07:52:32.458402 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.458349 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396304ef-8e18-48df-b0ab-d78b8c4afd46" containerName="maas-api" Sep 16 07:52:32.458402 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.458355 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="396304ef-8e18-48df-b0ab-d78b8c4afd46" containerName="maas-api" Sep 16 07:52:32.458521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.458429 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="083dde50-db57-461b-bd17-74e8f3335b10" containerName="maas-api" Sep 16 07:52:32.458521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.458442 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="396304ef-8e18-48df-b0ab-d78b8c4afd46" containerName="maas-api" Sep 16 07:52:32.463313 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.463285 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.466320 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.466298 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-4a243c-serving-cert\"" Sep 16 07:52:32.476353 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.476326 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr"] Sep 16 07:52:32.566862 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.566834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.566966 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.566888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.566966 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.566936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhnr\" (UniqueName: \"kubernetes.io/projected/5a0a69e1-a503-4145-b4c6-04784cbfff5c-kube-api-access-djhnr\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.567039 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.566980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a69e1-a503-4145-b4c6-04784cbfff5c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.667830 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.667800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.667956 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.667860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.667956 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.667886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhnr\" (UniqueName: \"kubernetes.io/projected/5a0a69e1-a503-4145-b4c6-04784cbfff5c-kube-api-access-djhnr\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.667956 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.667929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a69e1-a503-4145-b4c6-04784cbfff5c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.668115 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:52:32.667974 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-serving-cert: secret "maas-api-e2e-shared-a-w2-4a243c-serving-cert" not found Sep 16 07:52:32.668115 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:52:32.668041 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-maas-api-tls podName:5a0a69e1-a503-4145-b4c6-04784cbfff5c nodeName:}" failed. No retries permitted until 2026-09-16 07:52:33.16801956 +0000 UTC m=+1126.087635500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-maas-api-tls") pod "maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" (UID: "5a0a69e1-a503-4145-b4c6-04784cbfff5c") : secret "maas-api-e2e-shared-a-w2-4a243c-serving-cert" not found Sep 16 07:52:32.668604 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.668586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a69e1-a503-4145-b4c6-04784cbfff5c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.670143 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.670124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:32.677625 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:32.677603 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhnr\" (UniqueName: \"kubernetes.io/projected/5a0a69e1-a503-4145-b4c6-04784cbfff5c-kube-api-access-djhnr\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:33.172955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:33.172917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:33.175864 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:33.175834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:33.374900 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:33.374628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:33.656897 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:33.656873 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr"] Sep 16 07:52:33.657932 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:52:33.657905 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0a69e1_a503_4145_b4c6_04784cbfff5c.slice/crio-d214203c8edf96bf15a14c3201a821aa0d1c45cc324d32fdc73e08d7052facf2 WatchSource:0}: Error finding container d214203c8edf96bf15a14c3201a821aa0d1c45cc324d32fdc73e08d7052facf2: Status 404 returned error can't find the container with id d214203c8edf96bf15a14c3201a821aa0d1c45cc324d32fdc73e08d7052facf2 Sep 16 07:52:33.870838 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:33.870761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" event={"ID":"5a0a69e1-a503-4145-b4c6-04784cbfff5c","Type":"ContainerStarted","Data":"d214203c8edf96bf15a14c3201a821aa0d1c45cc324d32fdc73e08d7052facf2"} Sep 16 07:52:34.503313 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:34.503283 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:52:34.877897 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:34.877818 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" event={"ID":"5a0a69e1-a503-4145-b4c6-04784cbfff5c","Type":"ContainerStarted","Data":"b75a7e95e660bbb5c2aeb71fc7cf576f0dcfca80ae19e493c75e4360726e2650"} Sep 16 07:52:34.878281 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:34.877995 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:34.900899 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:34.900850 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" podStartSLOduration=2.418300955 podStartE2EDuration="2.900837937s" totalImagesPullingTime="482.536982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:52:34.897163408 +0000 UTC m=+1127.816779380" watchObservedRunningTime="2026-09-16 07:52:34.900837937 +0000 UTC m=+1127.820453883" Sep 16 07:52:40.890427 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:40.890392 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 07:52:53.414069 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:53.414034 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:52:55.792065 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.792031 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s"] Sep 16 07:52:55.797288 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.797261 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:55.801855 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.801751 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"distinct-w0-c9020d-kserve-self-signed-certs\"" Sep 16 07:52:55.801855 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.801799 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"kube-root-ca.crt\"" Sep 16 07:52:55.802048 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.801852 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"default-dockercfg-mcpgb\"" Sep 16 07:52:55.802116 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.802088 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"openshift-service-ca.crt\"" Sep 16 07:52:55.806920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.806896 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s"] Sep 16 07:52:55.967695 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.967661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/588048f5-a6c1-474b-bbac-c15065b7c5bb-tls-certs\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:55.967876 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.967704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-model-cache\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:55.967876 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.967855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-dshm\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:55.968004 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.967894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpt9\" (UniqueName: \"kubernetes.io/projected/588048f5-a6c1-474b-bbac-c15065b7c5bb-kube-api-access-6rpt9\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:55.968004 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.967941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-home\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:55.968004 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:55.967971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-tmp-dir\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.069200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.069112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-dshm\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.069200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.069146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpt9\" (UniqueName: \"kubernetes.io/projected/588048f5-a6c1-474b-bbac-c15065b7c5bb-kube-api-access-6rpt9\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.069200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.069174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-home\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.069200 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.069201 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-tmp-dir\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.069561 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.069287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/588048f5-a6c1-474b-bbac-c15065b7c5bb-tls-certs\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.069561 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.069325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-model-cache\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.069676 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.069631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-tmp-dir\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.069676 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.069650 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-home\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.069786 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.069681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-model-cache\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.071485 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.071456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-dshm\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.071622 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.071600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/588048f5-a6c1-474b-bbac-c15065b7c5bb-tls-certs\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.078534 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.078510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpt9\" (UniqueName: \"kubernetes.io/projected/588048f5-a6c1-474b-bbac-c15065b7c5bb-kube-api-access-6rpt9\") pod \"distinct-w0-c9020d-kserve-7b984dc56f-7f68s\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.107965 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.107948 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:52:56.383067 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:56.383041 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s"] Sep 16 07:52:56.384713 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:52:56.384688 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod588048f5_a6c1_474b_bbac_c15065b7c5bb.slice/crio-29619b016b6de2856d65cd1aec861f3a71a11dbc6b520e87fff7ac6183ae4b01 WatchSource:0}: Error finding container 29619b016b6de2856d65cd1aec861f3a71a11dbc6b520e87fff7ac6183ae4b01: Status 404 returned error can't find the container with id 29619b016b6de2856d65cd1aec861f3a71a11dbc6b520e87fff7ac6183ae4b01 Sep 16 07:52:57.009277 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:57.009236 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" event={"ID":"588048f5-a6c1-474b-bbac-c15065b7c5bb","Type":"ContainerStarted","Data":"6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55"} Sep 16 07:52:57.009676 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:57.009285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" event={"ID":"588048f5-a6c1-474b-bbac-c15065b7c5bb","Type":"ContainerStarted","Data":"29619b016b6de2856d65cd1aec861f3a71a11dbc6b520e87fff7ac6183ae4b01"} Sep 16 07:52:57.027853 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:52:57.027798 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" podStartSLOduration=2.027784434 podStartE2EDuration="2.027784434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:52:57.026060137 +0000 UTC m=+1149.945676107" watchObservedRunningTime="2026-09-16 07:52:57.027784434 +0000 UTC m=+1149.947400426" Sep 16 07:53:06.108476 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:06.108433 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:53:06.108861 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:06.108513 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:53:06.121278 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:06.121252 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:53:06.984326 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:06.984290 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh"] Sep 16 07:53:06.988439 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:06.988418 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:06.991629 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:06.991608 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-984c1a-serving-cert\"" Sep 16 07:53:07.000896 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.000871 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh"] Sep 16 07:53:07.083730 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.083703 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 07:53:07.162126 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.162097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpsc\" (UniqueName: \"kubernetes.io/projected/2ce7899e-5d2e-4e65-b611-b839f87ace2d-kube-api-access-xkpsc\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.162581 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.162195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce7899e-5d2e-4e65-b611-b839f87ace2d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.162581 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.162253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.162581 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.162352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.263406 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.263338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.263406 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.263398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpsc\" (UniqueName: \"kubernetes.io/projected/2ce7899e-5d2e-4e65-b611-b839f87ace2d-kube-api-access-xkpsc\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.263560 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.263439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce7899e-5d2e-4e65-b611-b839f87ace2d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.263560 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.263464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.263560 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:07.263472 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-serving-cert: secret "maas-api-e2e-shared-b-w2-984c1a-serving-cert" not found Sep 16 07:53:07.263560 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:07.263547 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-maas-api-tls podName:2ce7899e-5d2e-4e65-b611-b839f87ace2d nodeName:}" failed. No retries permitted until 2026-09-16 07:53:07.763529673 +0000 UTC m=+1160.683145599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-maas-api-tls") pod "maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" (UID: "2ce7899e-5d2e-4e65-b611-b839f87ace2d") : secret "maas-api-e2e-shared-b-w2-984c1a-serving-cert" not found Sep 16 07:53:07.264081 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.264055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce7899e-5d2e-4e65-b611-b839f87ace2d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.265773 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.265755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.277625 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.277604 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpsc\" (UniqueName: \"kubernetes.io/projected/2ce7899e-5d2e-4e65-b611-b839f87ace2d-kube-api-access-xkpsc\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.768384 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.768350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.771098 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.771069 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:07.901096 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:07.901057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:08.182852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:08.182821 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh"] Sep 16 07:53:08.183241 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:53:08.183178 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce7899e_5d2e_4e65_b611_b839f87ace2d.slice/crio-60f84cc7a2bbc9081e47867a2bbbe714759ed1acbe71f6311855d3ed501784f3 WatchSource:0}: Error finding container 60f84cc7a2bbc9081e47867a2bbbe714759ed1acbe71f6311855d3ed501784f3: Status 404 returned error can't find the container with id 60f84cc7a2bbc9081e47867a2bbbe714759ed1acbe71f6311855d3ed501784f3 Sep 16 07:53:09.085141 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:09.085108 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" event={"ID":"2ce7899e-5d2e-4e65-b611-b839f87ace2d","Type":"ContainerStarted","Data":"0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257"} Sep 16 07:53:09.085351 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:09.085158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" event={"ID":"2ce7899e-5d2e-4e65-b611-b839f87ace2d","Type":"ContainerStarted","Data":"60f84cc7a2bbc9081e47867a2bbbe714759ed1acbe71f6311855d3ed501784f3"} Sep 16 07:53:09.085351 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:09.085254 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:09.121984 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:09.121929 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" podStartSLOduration=2.841699154 podStartE2EDuration="3.121913558s" totalImagesPullingTime="280.214404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:09.119939184 +0000 UTC m=+1162.039555149" watchObservedRunningTime="2026-09-16 07:53:09.121913558 +0000 UTC m=+1162.041529504" Sep 16 07:53:15.097124 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:15.097039 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 07:53:16.122805 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:16.122773 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:53:33.106816 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:33.106780 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-64ce19"] Sep 16 07:53:33.110628 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:33.110608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-64ce19" Sep 16 07:53:33.113514 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:33.113493 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kvkmt\"" Sep 16 07:53:33.117350 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:33.117326 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-64ce19"] Sep 16 07:53:33.192545 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:33.192512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqxm\" (UniqueName: \"kubernetes.io/projected/4a2c99cf-4210-443a-92ac-deed4f1791dd-kube-api-access-cbqxm\") pod \"test-curl-7693-64ce19\" (UID: \"4a2c99cf-4210-443a-92ac-deed4f1791dd\") " pod="opendatahub/test-curl-7693-64ce19" Sep 16 07:53:33.293537 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:33.293500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqxm\" (UniqueName: \"kubernetes.io/projected/4a2c99cf-4210-443a-92ac-deed4f1791dd-kube-api-access-cbqxm\") pod \"test-curl-7693-64ce19\" (UID: \"4a2c99cf-4210-443a-92ac-deed4f1791dd\") " pod="opendatahub/test-curl-7693-64ce19" Sep 16 07:53:33.301887 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:33.301860 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqxm\" (UniqueName: \"kubernetes.io/projected/4a2c99cf-4210-443a-92ac-deed4f1791dd-kube-api-access-cbqxm\") pod \"test-curl-7693-64ce19\" (UID: \"4a2c99cf-4210-443a-92ac-deed4f1791dd\") " pod="opendatahub/test-curl-7693-64ce19" Sep 16 07:53:33.421622 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:33.421586 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-64ce19" Sep 16 07:53:33.691146 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:33.691119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-64ce19"] Sep 16 07:53:33.692662 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:53:33.692630 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a2c99cf_4210_443a_92ac_deed4f1791dd.slice/crio-165c662ae8da70bb33bf0196ca81653be6c79b249df332e2f494a19b5e57aba9 WatchSource:0}: Error finding container 165c662ae8da70bb33bf0196ca81653be6c79b249df332e2f494a19b5e57aba9: Status 404 returned error can't find the container with id 165c662ae8da70bb33bf0196ca81653be6c79b249df332e2f494a19b5e57aba9 Sep 16 07:53:34.237796 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.237763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-64ce19" event={"ID":"4a2c99cf-4210-443a-92ac-deed4f1791dd","Type":"ContainerStarted","Data":"50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6"} Sep 16 07:53:34.238169 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.237806 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-64ce19" event={"ID":"4a2c99cf-4210-443a-92ac-deed4f1791dd","Type":"ContainerStarted","Data":"165c662ae8da70bb33bf0196ca81653be6c79b249df332e2f494a19b5e57aba9"} Sep 16 07:53:34.257018 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.256967 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-64ce19" podStartSLOduration=1.256953341 podStartE2EDuration="1.256953341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:34.25515789 +0000 UTC m=+1187.174773837" watchObservedRunningTime="2026-09-16 07:53:34.256953341 +0000 UTC m=+1187.176569286" Sep 16 07:53:34.819010 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.818974 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-64ce19"] Sep 16 07:53:34.823587 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.823560 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-64ce19"] Sep 16 07:53:34.936386 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.936357 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-e7179a"] Sep 16 07:53:34.936852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.936836 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2c99cf-4210-443a-92ac-deed4f1791dd" containerName="test-curl-7693-64ce19" Sep 16 07:53:34.936901 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.936855 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2c99cf-4210-443a-92ac-deed4f1791dd" containerName="test-curl-7693-64ce19" Sep 16 07:53:34.936948 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.936938 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2c99cf-4210-443a-92ac-deed4f1791dd" containerName="test-curl-7693-64ce19" Sep 16 07:53:34.940373 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.940357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-e7179a" Sep 16 07:53:34.946134 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:34.946110 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-e7179a"] Sep 16 07:53:35.007770 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:35.007749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8twb\" (UniqueName: \"kubernetes.io/projected/3bbf0be7-37ec-4078-adea-8aac448091c6-kube-api-access-c8twb\") pod \"test-curl-7693-e7179a\" (UID: \"3bbf0be7-37ec-4078-adea-8aac448091c6\") " pod="opendatahub/test-curl-7693-e7179a" Sep 16 07:53:35.108666 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:35.108641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8twb\" (UniqueName: \"kubernetes.io/projected/3bbf0be7-37ec-4078-adea-8aac448091c6-kube-api-access-c8twb\") pod \"test-curl-7693-e7179a\" (UID: \"3bbf0be7-37ec-4078-adea-8aac448091c6\") " pod="opendatahub/test-curl-7693-e7179a" Sep 16 07:53:35.117123 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:35.117100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8twb\" (UniqueName: \"kubernetes.io/projected/3bbf0be7-37ec-4078-adea-8aac448091c6-kube-api-access-c8twb\") pod \"test-curl-7693-e7179a\" (UID: \"3bbf0be7-37ec-4078-adea-8aac448091c6\") " pod="opendatahub/test-curl-7693-e7179a" Sep 16 07:53:35.243467 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:35.243410 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7693-64ce19" container="test-curl-7693-64ce19" Sep 16 07:53:35.252066 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:35.252043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-e7179a" Sep 16 07:53:35.524564 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:35.524536 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-e7179a"] Sep 16 07:53:35.524885 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:53:35.524858 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bbf0be7_37ec_4078_adea_8aac448091c6.slice/crio-a450d967a7e28b6ac146a8b179a861244d5f56211b2378f001a9b7e73141476a WatchSource:0}: Error finding container a450d967a7e28b6ac146a8b179a861244d5f56211b2378f001a9b7e73141476a: Status 404 returned error can't find the container with id a450d967a7e28b6ac146a8b179a861244d5f56211b2378f001a9b7e73141476a Sep 16 07:53:36.251525 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.251485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-e7179a" event={"ID":"3bbf0be7-37ec-4078-adea-8aac448091c6","Type":"ContainerStarted","Data":"a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312"} Sep 16 07:53:36.252007 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.251535 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-e7179a" event={"ID":"3bbf0be7-37ec-4078-adea-8aac448091c6","Type":"ContainerStarted","Data":"a450d967a7e28b6ac146a8b179a861244d5f56211b2378f001a9b7e73141476a"} Sep 16 07:53:36.252007 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.251652 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-64ce19" podUID="4a2c99cf-4210-443a-92ac-deed4f1791dd" containerName="test-curl-7693-64ce19" containerID="cri-o://50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6" gracePeriod=2 Sep 16 07:53:36.267742 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.267699 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-e7179a" podStartSLOduration=2.267683018 podStartE2EDuration="2.267683018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:36.265740412 +0000 UTC m=+1189.185356358" watchObservedRunningTime="2026-09-16 07:53:36.267683018 +0000 UTC m=+1189.187298963" Sep 16 07:53:36.855702 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.855656 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-e7179a"] Sep 16 07:53:36.862232 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.862189 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-e7179a"] Sep 16 07:53:36.970403 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.970373 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-b04c74"] Sep 16 07:53:36.970855 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.970841 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bbf0be7-37ec-4078-adea-8aac448091c6" containerName="test-curl-7693-e7179a" Sep 16 07:53:36.970912 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.970877 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bbf0be7-37ec-4078-adea-8aac448091c6" containerName="test-curl-7693-e7179a" Sep 16 07:53:36.970966 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.970957 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bbf0be7-37ec-4078-adea-8aac448091c6" containerName="test-curl-7693-e7179a" Sep 16 07:53:36.974131 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.974113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-b04c74" Sep 16 07:53:36.980866 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:36.980842 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-b04c74"] Sep 16 07:53:37.025411 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:37.025372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8nx6\" (UniqueName: \"kubernetes.io/projected/f24922ca-af6b-4578-b30f-e54e04d044fc-kube-api-access-n8nx6\") pod \"test-curl-7693-b04c74\" (UID: \"f24922ca-af6b-4578-b30f-e54e04d044fc\") " pod="opendatahub/test-curl-7693-b04c74" Sep 16 07:53:37.126694 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:37.126653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8nx6\" (UniqueName: \"kubernetes.io/projected/f24922ca-af6b-4578-b30f-e54e04d044fc-kube-api-access-n8nx6\") pod \"test-curl-7693-b04c74\" (UID: \"f24922ca-af6b-4578-b30f-e54e04d044fc\") " pod="opendatahub/test-curl-7693-b04c74" Sep 16 07:53:37.135070 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:37.135044 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8nx6\" (UniqueName: \"kubernetes.io/projected/f24922ca-af6b-4578-b30f-e54e04d044fc-kube-api-access-n8nx6\") pod \"test-curl-7693-b04c74\" (UID: \"f24922ca-af6b-4578-b30f-e54e04d044fc\") " pod="opendatahub/test-curl-7693-b04c74" Sep 16 07:53:37.258001 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:37.257927 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7693-e7179a" container="test-curl-7693-e7179a" Sep 16 07:53:37.286493 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:37.286464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-b04c74" Sep 16 07:53:37.561073 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:53:37.561044 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf24922ca_af6b_4578_b30f_e54e04d044fc.slice/crio-df6d241f852d4d6344539804378e82ea70b57f98aacd5eaf597cf54153b8e097 WatchSource:0}: Error finding container df6d241f852d4d6344539804378e82ea70b57f98aacd5eaf597cf54153b8e097: Status 404 returned error can't find the container with id df6d241f852d4d6344539804378e82ea70b57f98aacd5eaf597cf54153b8e097 Sep 16 07:53:37.562773 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:37.562669 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-b04c74"] Sep 16 07:53:38.265834 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.265798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-b04c74" event={"ID":"f24922ca-af6b-4578-b30f-e54e04d044fc","Type":"ContainerStarted","Data":"7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94"} Sep 16 07:53:38.266243 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.265844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-b04c74" event={"ID":"f24922ca-af6b-4578-b30f-e54e04d044fc","Type":"ContainerStarted","Data":"df6d241f852d4d6344539804378e82ea70b57f98aacd5eaf597cf54153b8e097"} Sep 16 07:53:38.266243 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.265957 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-e7179a" podUID="3bbf0be7-37ec-4078-adea-8aac448091c6" containerName="test-curl-7693-e7179a" containerID="cri-o://a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312" gracePeriod=2 Sep 16 07:53:38.279768 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.279601 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-b04c74" podStartSLOduration=2.279582736 podStartE2EDuration="2.279582736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:38.279477696 +0000 UTC m=+1191.199093643" watchObservedRunningTime="2026-09-16 07:53:38.279582736 +0000 UTC m=+1191.199198683" Sep 16 07:53:38.436635 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.436609 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-64ce19" Sep 16 07:53:38.540011 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.539947 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbqxm\" (UniqueName: \"kubernetes.io/projected/4a2c99cf-4210-443a-92ac-deed4f1791dd-kube-api-access-cbqxm\") pod \"4a2c99cf-4210-443a-92ac-deed4f1791dd\" (UID: \"4a2c99cf-4210-443a-92ac-deed4f1791dd\") " Sep 16 07:53:38.541817 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.541792 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2c99cf-4210-443a-92ac-deed4f1791dd-kube-api-access-cbqxm" (OuterVolumeSpecName: "kube-api-access-cbqxm") pod "4a2c99cf-4210-443a-92ac-deed4f1791dd" (UID: "4a2c99cf-4210-443a-92ac-deed4f1791dd"). InnerVolumeSpecName "kube-api-access-cbqxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:53:38.641439 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.641159 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbqxm\" (UniqueName: \"kubernetes.io/projected/4a2c99cf-4210-443a-92ac-deed4f1791dd-kube-api-access-cbqxm\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:53:38.849920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.849850 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-b04c74"] Sep 16 07:53:38.854544 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.854516 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-b04c74"] Sep 16 07:53:38.961831 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.961797 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-5704e5"] Sep 16 07:53:38.962296 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.962266 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f24922ca-af6b-4578-b30f-e54e04d044fc" containerName="test-curl-7693-b04c74" Sep 16 07:53:38.962339 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.962300 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f24922ca-af6b-4578-b30f-e54e04d044fc" containerName="test-curl-7693-b04c74" Sep 16 07:53:38.962407 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.962397 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f24922ca-af6b-4578-b30f-e54e04d044fc" containerName="test-curl-7693-b04c74" Sep 16 07:53:38.965518 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.965502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-5704e5" Sep 16 07:53:38.971278 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:38.971255 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-5704e5"] Sep 16 07:53:39.044955 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.044929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6f9d\" (UniqueName: \"kubernetes.io/projected/bae3ce2d-d62c-479f-9df0-89ea7a45fc30-kube-api-access-c6f9d\") pod \"test-curl-7693-5704e5\" (UID: \"bae3ce2d-d62c-479f-9df0-89ea7a45fc30\") " pod="opendatahub/test-curl-7693-5704e5" Sep 16 07:53:39.145956 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.145902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6f9d\" (UniqueName: \"kubernetes.io/projected/bae3ce2d-d62c-479f-9df0-89ea7a45fc30-kube-api-access-c6f9d\") pod \"test-curl-7693-5704e5\" (UID: \"bae3ce2d-d62c-479f-9df0-89ea7a45fc30\") " pod="opendatahub/test-curl-7693-5704e5" Sep 16 07:53:39.154162 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.154144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6f9d\" (UniqueName: \"kubernetes.io/projected/bae3ce2d-d62c-479f-9df0-89ea7a45fc30-kube-api-access-c6f9d\") pod \"test-curl-7693-5704e5\" (UID: \"bae3ce2d-d62c-479f-9df0-89ea7a45fc30\") " pod="opendatahub/test-curl-7693-5704e5" Sep 16 07:53:39.272519 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.272491 2580 generic.go:356] "Generic (PLEG): container finished" podID="4a2c99cf-4210-443a-92ac-deed4f1791dd" containerID="50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6" exitCode=137 Sep 16 07:53:39.272859 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.272534 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-64ce19" Sep 16 07:53:39.272859 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.272596 2580 scope.go:117] "RemoveContainer" containerID="50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6" Sep 16 07:53:39.272859 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:39.272805 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7693-b04c74" container="test-curl-7693-b04c74" Sep 16 07:53:39.276997 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.276971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-5704e5" Sep 16 07:53:39.282056 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.282042 2580 scope.go:117] "RemoveContainer" containerID="50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6" Sep 16 07:53:39.282318 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:39.282296 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6\": container with ID starting with 50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6 not found: ID does not exist" containerID="50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6" Sep 16 07:53:39.282406 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.282328 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6"} err="failed to get container status \"50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6\": rpc error: code = NotFound desc = could not find container \"50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6\": container with ID starting with 50e2df1327cc0df026fe69181c02731813f5f96e9825d224711533420b9291c6 not found: ID does not exist" Sep 16 07:53:39.283829 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:39.283804 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-64ce19\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="4a2c99cf-4210-443a-92ac-deed4f1791dd" pod="opendatahub/test-curl-7693-64ce19" Sep 16 07:53:39.558003 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:53:39.557969 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbae3ce2d_d62c_479f_9df0_89ea7a45fc30.slice/crio-ee49e6d299cfa89315e156f5230912d2a4e951de00c99476a0243a5b8a77b845 WatchSource:0}: Error finding container ee49e6d299cfa89315e156f5230912d2a4e951de00c99476a0243a5b8a77b845: Status 404 returned error can't find the container with id ee49e6d299cfa89315e156f5230912d2a4e951de00c99476a0243a5b8a77b845 Sep 16 07:53:39.561781 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.561757 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a2c99cf-4210-443a-92ac-deed4f1791dd" path="/var/lib/kubelet/pods/4a2c99cf-4210-443a-92ac-deed4f1791dd/volumes" Sep 16 07:53:39.562054 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:39.562040 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-5704e5"] Sep 16 07:53:40.282301 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:40.282259 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-5704e5" event={"ID":"bae3ce2d-d62c-479f-9df0-89ea7a45fc30","Type":"ContainerStarted","Data":"97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26"} Sep 16 07:53:40.282661 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:40.282325 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-5704e5" event={"ID":"bae3ce2d-d62c-479f-9df0-89ea7a45fc30","Type":"ContainerStarted","Data":"ee49e6d299cfa89315e156f5230912d2a4e951de00c99476a0243a5b8a77b845"} Sep 16 07:53:40.282661 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:40.282468 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-b04c74" podUID="f24922ca-af6b-4578-b30f-e54e04d044fc" containerName="test-curl-7693-b04c74" containerID="cri-o://7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94" gracePeriod=2 Sep 16 07:53:40.297967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:40.297925 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-5704e5" podStartSLOduration=2.297910763 podStartE2EDuration="2.297910763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:40.295454611 +0000 UTC m=+1193.215070556" watchObservedRunningTime="2026-09-16 07:53:40.297910763 +0000 UTC m=+1193.217526759" Sep 16 07:53:40.461025 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:40.461001 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-e7179a" Sep 16 07:53:40.558195 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:40.558129 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8twb\" (UniqueName: \"kubernetes.io/projected/3bbf0be7-37ec-4078-adea-8aac448091c6-kube-api-access-c8twb\") pod \"3bbf0be7-37ec-4078-adea-8aac448091c6\" (UID: \"3bbf0be7-37ec-4078-adea-8aac448091c6\") " Sep 16 07:53:40.559918 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:40.559889 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bbf0be7-37ec-4078-adea-8aac448091c6-kube-api-access-c8twb" (OuterVolumeSpecName: "kube-api-access-c8twb") pod "3bbf0be7-37ec-4078-adea-8aac448091c6" (UID: "3bbf0be7-37ec-4078-adea-8aac448091c6"). InnerVolumeSpecName "kube-api-access-c8twb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:53:40.659001 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:40.658971 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8twb\" (UniqueName: \"kubernetes.io/projected/3bbf0be7-37ec-4078-adea-8aac448091c6-kube-api-access-c8twb\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:53:40.869801 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:40.869715 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-5704e5"] Sep 16 07:53:40.875170 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:40.875141 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-5704e5"] Sep 16 07:53:41.290036 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:41.290004 2580 generic.go:356] "Generic (PLEG): container finished" podID="3bbf0be7-37ec-4078-adea-8aac448091c6" containerID="a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312" exitCode=137 Sep 16 07:53:41.290459 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:41.290047 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-e7179a" Sep 16 07:53:41.290459 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:41.290093 2580 scope.go:117] "RemoveContainer" containerID="a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312" Sep 16 07:53:41.290459 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:41.290376 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7693-5704e5" container="test-curl-7693-5704e5" Sep 16 07:53:41.302516 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:41.302498 2580 scope.go:117] "RemoveContainer" containerID="a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312" Sep 16 07:53:41.302705 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:41.302684 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-e7179a\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="3bbf0be7-37ec-4078-adea-8aac448091c6" pod="opendatahub/test-curl-7693-e7179a" Sep 16 07:53:41.302785 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:41.302767 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312\": container with ID starting with a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312 not found: ID does not exist" containerID="a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312" Sep 16 07:53:41.302826 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:41.302795 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312"} err="failed to get container status \"a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312\": rpc error: code = NotFound desc = could not find container \"a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312\": container with ID starting with a16374092162e1c6084d388cd07a05312ae29861c2fff44da9795bfff542a312 not found: ID does not exist" Sep 16 07:53:41.561169 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:41.561099 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bbf0be7-37ec-4078-adea-8aac448091c6" path="/var/lib/kubelet/pods/3bbf0be7-37ec-4078-adea-8aac448091c6/volumes" Sep 16 07:53:42.297481 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:42.297453 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-5704e5" podUID="bae3ce2d-d62c-479f-9df0-89ea7a45fc30" containerName="test-curl-7693-5704e5" containerID="cri-o://97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26" gracePeriod=2 Sep 16 07:53:42.467426 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:42.467398 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-b04c74" Sep 16 07:53:42.577808 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:42.577734 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8nx6\" (UniqueName: \"kubernetes.io/projected/f24922ca-af6b-4578-b30f-e54e04d044fc-kube-api-access-n8nx6\") pod \"f24922ca-af6b-4578-b30f-e54e04d044fc\" (UID: \"f24922ca-af6b-4578-b30f-e54e04d044fc\") " Sep 16 07:53:42.579818 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:42.579792 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24922ca-af6b-4578-b30f-e54e04d044fc-kube-api-access-n8nx6" (OuterVolumeSpecName: "kube-api-access-n8nx6") pod "f24922ca-af6b-4578-b30f-e54e04d044fc" (UID: "f24922ca-af6b-4578-b30f-e54e04d044fc"). InnerVolumeSpecName "kube-api-access-n8nx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:53:42.678586 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:42.678563 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8nx6\" (UniqueName: \"kubernetes.io/projected/f24922ca-af6b-4578-b30f-e54e04d044fc-kube-api-access-n8nx6\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:53:43.304543 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:43.304504 2580 generic.go:356] "Generic (PLEG): container finished" podID="f24922ca-af6b-4578-b30f-e54e04d044fc" containerID="7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94" exitCode=137 Sep 16 07:53:43.304543 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:43.304547 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-b04c74" Sep 16 07:53:43.304990 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:43.304609 2580 scope.go:117] "RemoveContainer" containerID="7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94" Sep 16 07:53:43.314780 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:43.314757 2580 scope.go:117] "RemoveContainer" containerID="7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94" Sep 16 07:53:43.315069 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:43.315049 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94\": container with ID starting with 7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94 not found: ID does not exist" containerID="7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94" Sep 16 07:53:43.315128 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:43.315075 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94"} err="failed to get container status \"7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94\": rpc error: code = NotFound desc = could not find container \"7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94\": container with ID starting with 7e125e1678b450e1329bbdfcdae43f61e820f41240f91446aa8792e1ed548a94 not found: ID does not exist" Sep 16 07:53:43.315455 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:43.315430 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-b04c74\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="f24922ca-af6b-4578-b30f-e54e04d044fc" pod="opendatahub/test-curl-7693-b04c74" Sep 16 07:53:43.561800 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:43.561704 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f24922ca-af6b-4578-b30f-e54e04d044fc" path="/var/lib/kubelet/pods/f24922ca-af6b-4578-b30f-e54e04d044fc/volumes" Sep 16 07:53:44.486960 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:44.486937 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-5704e5" Sep 16 07:53:44.596600 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:44.596538 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6f9d\" (UniqueName: \"kubernetes.io/projected/bae3ce2d-d62c-479f-9df0-89ea7a45fc30-kube-api-access-c6f9d\") pod \"bae3ce2d-d62c-479f-9df0-89ea7a45fc30\" (UID: \"bae3ce2d-d62c-479f-9df0-89ea7a45fc30\") " Sep 16 07:53:44.598607 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:44.598584 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae3ce2d-d62c-479f-9df0-89ea7a45fc30-kube-api-access-c6f9d" (OuterVolumeSpecName: "kube-api-access-c6f9d") pod "bae3ce2d-d62c-479f-9df0-89ea7a45fc30" (UID: "bae3ce2d-d62c-479f-9df0-89ea7a45fc30"). InnerVolumeSpecName "kube-api-access-c6f9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:53:44.697971 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:44.697825 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6f9d\" (UniqueName: \"kubernetes.io/projected/bae3ce2d-d62c-479f-9df0-89ea7a45fc30-kube-api-access-c6f9d\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:53:45.320202 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:45.320166 2580 generic.go:356] "Generic (PLEG): container finished" podID="bae3ce2d-d62c-479f-9df0-89ea7a45fc30" containerID="97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26" exitCode=137 Sep 16 07:53:45.320382 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:45.320205 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-5704e5" Sep 16 07:53:45.320382 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:45.320250 2580 scope.go:117] "RemoveContainer" containerID="97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26" Sep 16 07:53:45.329870 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:45.329850 2580 scope.go:117] "RemoveContainer" containerID="97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26" Sep 16 07:53:45.330141 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:45.330123 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26\": container with ID starting with 97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26 not found: ID does not exist" containerID="97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26" Sep 16 07:53:45.330178 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:45.330149 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26"} err="failed to get container status \"97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26\": rpc error: code = NotFound desc = could not find container \"97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26\": container with ID starting with 97f632ceac60b59a4dfb10dea56abeaa3fe64081bdbe43c68c4ddf5b716b5e26 not found: ID does not exist" Sep 16 07:53:45.330756 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:45.330735 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-5704e5\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="bae3ce2d-d62c-479f-9df0-89ea7a45fc30" pod="opendatahub/test-curl-7693-5704e5" Sep 16 07:53:45.561070 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:45.561041 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bae3ce2d-d62c-479f-9df0-89ea7a45fc30" path="/var/lib/kubelet/pods/bae3ce2d-d62c-479f-9df0-89ea7a45fc30/volumes" Sep 16 07:53:47.546669 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:47.546638 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:53:47.549920 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:47.549894 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:53:48.578232 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.578178 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t"] Sep 16 07:53:48.578719 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.578703 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bae3ce2d-d62c-479f-9df0-89ea7a45fc30" containerName="test-curl-7693-5704e5" Sep 16 07:53:48.578773 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.578720 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bae3ce2d-d62c-479f-9df0-89ea7a45fc30" containerName="test-curl-7693-5704e5" Sep 16 07:53:48.578814 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.578796 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bae3ce2d-d62c-479f-9df0-89ea7a45fc30" containerName="test-curl-7693-5704e5" Sep 16 07:53:48.583834 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.583807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.586596 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.586572 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fc2a294d-serving-cert\"" Sep 16 07:53:48.590724 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.590694 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t"] Sep 16 07:53:48.735305 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.735274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-maas-api-tls\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.735457 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.735309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc24c\" (UniqueName: \"kubernetes.io/projected/904d5328-61af-45a7-8fc5-ef936932733a-kube-api-access-cc24c\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.735457 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.735333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-metrics-certs\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.735457 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.735443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904d5328-61af-45a7-8fc5-ef936932733a-openshift-service-ca\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.835877 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.835804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904d5328-61af-45a7-8fc5-ef936932733a-openshift-service-ca\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.835877 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.835860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-maas-api-tls\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.836048 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.835881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc24c\" (UniqueName: \"kubernetes.io/projected/904d5328-61af-45a7-8fc5-ef936932733a-kube-api-access-cc24c\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.836048 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.835906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-metrics-certs\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.836048 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:48.836009 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-serving-cert: secret "maas-api-e2e-mt-fc2a294d-serving-cert" not found Sep 16 07:53:48.836163 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:53:48.836085 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-maas-api-tls podName:904d5328-61af-45a7-8fc5-ef936932733a nodeName:}" failed. No retries permitted until 2026-09-16 07:53:49.336061682 +0000 UTC m=+1202.255677629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-maas-api-tls") pod "maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" (UID: "904d5328-61af-45a7-8fc5-ef936932733a") : secret "maas-api-e2e-mt-fc2a294d-serving-cert" not found Sep 16 07:53:48.836612 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.836581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904d5328-61af-45a7-8fc5-ef936932733a-openshift-service-ca\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.838380 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.838350 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-metrics-certs\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:48.858613 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:48.858592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc24c\" (UniqueName: \"kubernetes.io/projected/904d5328-61af-45a7-8fc5-ef936932733a-kube-api-access-cc24c\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:49.340525 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:49.340495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-maas-api-tls\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:49.342982 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:49.342948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-maas-api-tls\") pod \"maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:49.497204 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:49.497166 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:49.770780 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:49.770749 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t"] Sep 16 07:53:49.771816 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:53:49.771793 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904d5328_61af_45a7_8fc5_ef936932733a.slice/crio-0e34915128e01c7367622ec0ba8ea19d6bcb4a74baeeb0b937e73333377b1de1 WatchSource:0}: Error finding container 0e34915128e01c7367622ec0ba8ea19d6bcb4a74baeeb0b937e73333377b1de1: Status 404 returned error can't find the container with id 0e34915128e01c7367622ec0ba8ea19d6bcb4a74baeeb0b937e73333377b1de1 Sep 16 07:53:50.354569 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:50.354535 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" event={"ID":"904d5328-61af-45a7-8fc5-ef936932733a","Type":"ContainerStarted","Data":"30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2"} Sep 16 07:53:50.354762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:50.354593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" event={"ID":"904d5328-61af-45a7-8fc5-ef936932733a","Type":"ContainerStarted","Data":"0e34915128e01c7367622ec0ba8ea19d6bcb4a74baeeb0b937e73333377b1de1"} Sep 16 07:53:50.354762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:50.354644 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:50.377489 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:50.377442 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" podStartSLOduration=2.004000298 podStartE2EDuration="2.37742761s" totalImagesPullingTime="373.427312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:50.374907251 +0000 UTC m=+1203.294523197" watchObservedRunningTime="2026-09-16 07:53:50.37742761 +0000 UTC m=+1203.297043556" Sep 16 07:53:56.366227 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:56.366184 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:53:56.705188 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:53:56.705144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:54:17.247127 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:17.247096 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn"] Sep 16 07:54:22.833701 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:22.833645 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj"] Sep 16 07:54:22.836322 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:22.833959 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" podUID="92611524-dc02-4c2f-a048-ce8570aaada8" containerName="maas-api" containerID="cri-o://2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39" gracePeriod=30 Sep 16 07:54:23.103709 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.103678 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:54:23.241717 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.241688 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc9dp\" (UniqueName: \"kubernetes.io/projected/92611524-dc02-4c2f-a048-ce8570aaada8-kube-api-access-pc9dp\") pod \"92611524-dc02-4c2f-a048-ce8570aaada8\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " Sep 16 07:54:23.241869 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.241754 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-metrics-certs\") pod \"92611524-dc02-4c2f-a048-ce8570aaada8\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " Sep 16 07:54:23.241869 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.241816 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92611524-dc02-4c2f-a048-ce8570aaada8-openshift-service-ca\") pod \"92611524-dc02-4c2f-a048-ce8570aaada8\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " Sep 16 07:54:23.241978 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.241960 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-maas-api-tls\") pod \"92611524-dc02-4c2f-a048-ce8570aaada8\" (UID: \"92611524-dc02-4c2f-a048-ce8570aaada8\") " Sep 16 07:54:23.242305 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.242271 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92611524-dc02-4c2f-a048-ce8570aaada8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "92611524-dc02-4c2f-a048-ce8570aaada8" (UID: "92611524-dc02-4c2f-a048-ce8570aaada8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:54:23.242453 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.242402 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92611524-dc02-4c2f-a048-ce8570aaada8-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:23.243958 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.243928 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "92611524-dc02-4c2f-a048-ce8570aaada8" (UID: "92611524-dc02-4c2f-a048-ce8570aaada8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:54:23.244059 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.243954 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92611524-dc02-4c2f-a048-ce8570aaada8-kube-api-access-pc9dp" (OuterVolumeSpecName: "kube-api-access-pc9dp") pod "92611524-dc02-4c2f-a048-ce8570aaada8" (UID: "92611524-dc02-4c2f-a048-ce8570aaada8"). InnerVolumeSpecName "kube-api-access-pc9dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:54:23.244059 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.243953 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "92611524-dc02-4c2f-a048-ce8570aaada8" (UID: "92611524-dc02-4c2f-a048-ce8570aaada8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:54:23.343324 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.343263 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc9dp\" (UniqueName: \"kubernetes.io/projected/92611524-dc02-4c2f-a048-ce8570aaada8-kube-api-access-pc9dp\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:23.343324 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.343286 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:23.343324 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.343296 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92611524-dc02-4c2f-a048-ce8570aaada8-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:23.555430 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.555397 2580 generic.go:356] "Generic (PLEG): container finished" podID="92611524-dc02-4c2f-a048-ce8570aaada8" containerID="2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39" exitCode=0 Sep 16 07:54:23.555569 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.555497 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" Sep 16 07:54:23.561985 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.561956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" event={"ID":"92611524-dc02-4c2f-a048-ce8570aaada8","Type":"ContainerDied","Data":"2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39"} Sep 16 07:54:23.562100 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.562008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj" event={"ID":"92611524-dc02-4c2f-a048-ce8570aaada8","Type":"ContainerDied","Data":"e7b3394b1a170f7fab8656371bd288dec2afff349a8e25e1ddb96bbe43035004"} Sep 16 07:54:23.562100 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.562040 2580 scope.go:117] "RemoveContainer" containerID="2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39" Sep 16 07:54:23.571750 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.571734 2580 scope.go:117] "RemoveContainer" containerID="2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39" Sep 16 07:54:23.572014 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:54:23.571997 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39\": container with ID starting with 2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39 not found: ID does not exist" containerID="2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39" Sep 16 07:54:23.572062 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.572023 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39"} err="failed to get container status \"2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39\": rpc error: code = NotFound desc = could not find container \"2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39\": container with ID starting with 2bae40fb99b9916de1d8ac2aff9e1fef6375320c8ab21c74185ed5b805064d39 not found: ID does not exist" Sep 16 07:54:23.581591 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.581572 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj"] Sep 16 07:54:23.585387 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:23.585363 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5dd30f-755cb58f6-tsscj"] Sep 16 07:54:25.560500 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:25.560470 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92611524-dc02-4c2f-a048-ce8570aaada8" path="/var/lib/kubelet/pods/92611524-dc02-4c2f-a048-ce8570aaada8/volumes" Sep 16 07:54:29.902342 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:29.902308 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:54:32.267261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.267202 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" podUID="1b02c97c-ca40-4aed-9814-54e40eceadd2" containerName="main" containerID="cri-o://1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87" gracePeriod=45 Sep 16 07:54:32.516890 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.516865 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:54:32.609692 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.609608 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b02c97c-ca40-4aed-9814-54e40eceadd2" containerID="1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87" exitCode=0 Sep 16 07:54:32.609692 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.609653 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" Sep 16 07:54:32.609692 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.609677 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" event={"ID":"1b02c97c-ca40-4aed-9814-54e40eceadd2","Type":"ContainerDied","Data":"1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87"} Sep 16 07:54:32.609921 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.609714 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn" event={"ID":"1b02c97c-ca40-4aed-9814-54e40eceadd2","Type":"ContainerDied","Data":"858810ab777ffa6eea47922178ea5df3c1cf0c86e54c488e7edef9594775bde1"} Sep 16 07:54:32.609921 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.609736 2580 scope.go:117] "RemoveContainer" containerID="1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87" Sep 16 07:54:32.618762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.618741 2580 scope.go:117] "RemoveContainer" containerID="1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87" Sep 16 07:54:32.619021 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:54:32.619002 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87\": container with ID starting with 1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87 not found: ID does not exist" containerID="1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87" Sep 16 07:54:32.619075 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.619031 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87"} err="failed to get container status \"1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87\": rpc error: code = NotFound desc = could not find container \"1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87\": container with ID starting with 1fdd1a459e170b3129d22e47ce206eb974b663df52c00f038c36332fc0b77d87 not found: ID does not exist" Sep 16 07:54:32.626347 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.626326 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-home\") pod \"1b02c97c-ca40-4aed-9814-54e40eceadd2\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " Sep 16 07:54:32.626451 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.626439 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhwp4\" (UniqueName: \"kubernetes.io/projected/1b02c97c-ca40-4aed-9814-54e40eceadd2-kube-api-access-nhwp4\") pod \"1b02c97c-ca40-4aed-9814-54e40eceadd2\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " Sep 16 07:54:32.626512 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.626485 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b02c97c-ca40-4aed-9814-54e40eceadd2-tls-certs\") pod \"1b02c97c-ca40-4aed-9814-54e40eceadd2\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " Sep 16 07:54:32.626569 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.626551 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-model-cache\") pod \"1b02c97c-ca40-4aed-9814-54e40eceadd2\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " Sep 16 07:54:32.626620 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.626574 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-home" (OuterVolumeSpecName: "home") pod "1b02c97c-ca40-4aed-9814-54e40eceadd2" (UID: "1b02c97c-ca40-4aed-9814-54e40eceadd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:54:32.626620 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.626580 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-dshm\") pod \"1b02c97c-ca40-4aed-9814-54e40eceadd2\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " Sep 16 07:54:32.626726 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.626653 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-tmp-dir\") pod \"1b02c97c-ca40-4aed-9814-54e40eceadd2\" (UID: \"1b02c97c-ca40-4aed-9814-54e40eceadd2\") " Sep 16 07:54:32.626902 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.626854 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b02c97c-ca40-4aed-9814-54e40eceadd2" (UID: "1b02c97c-ca40-4aed-9814-54e40eceadd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:54:32.626902 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.626889 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b02c97c-ca40-4aed-9814-54e40eceadd2" (UID: "1b02c97c-ca40-4aed-9814-54e40eceadd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:54:32.627080 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.627057 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-model-cache\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:32.627080 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.627077 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-tmp-dir\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:32.627174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.627090 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-home\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:32.628628 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.628604 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b02c97c-ca40-4aed-9814-54e40eceadd2-kube-api-access-nhwp4" (OuterVolumeSpecName: "kube-api-access-nhwp4") pod "1b02c97c-ca40-4aed-9814-54e40eceadd2" (UID: "1b02c97c-ca40-4aed-9814-54e40eceadd2"). InnerVolumeSpecName "kube-api-access-nhwp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:54:32.628708 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.628611 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-dshm" (OuterVolumeSpecName: "dshm") pod "1b02c97c-ca40-4aed-9814-54e40eceadd2" (UID: "1b02c97c-ca40-4aed-9814-54e40eceadd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:54:32.628708 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.628635 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b02c97c-ca40-4aed-9814-54e40eceadd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b02c97c-ca40-4aed-9814-54e40eceadd2" (UID: "1b02c97c-ca40-4aed-9814-54e40eceadd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:54:32.728224 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.728178 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhwp4\" (UniqueName: \"kubernetes.io/projected/1b02c97c-ca40-4aed-9814-54e40eceadd2-kube-api-access-nhwp4\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:32.728340 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.728206 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b02c97c-ca40-4aed-9814-54e40eceadd2-tls-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:32.728340 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.728246 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b02c97c-ca40-4aed-9814-54e40eceadd2-dshm\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:32.931873 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.931849 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn"] Sep 16 07:54:32.935333 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:32.935311 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5dd30f/test-model-5dd30f-kserve-75987f849c-mncsn"] Sep 16 07:54:33.560929 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:33.560896 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b02c97c-ca40-4aed-9814-54e40eceadd2" path="/var/lib/kubelet/pods/1b02c97c-ca40-4aed-9814-54e40eceadd2/volumes" Sep 16 07:54:45.477818 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.477710 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz"] Sep 16 07:54:45.478269 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.478001 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" podUID="d2e6987a-0fa0-48ad-8101-9fc027c576e6" containerName="maas-api" containerID="cri-o://0d89814f794c47f70041111caa69d076d476ff2afd08ec9aacecbadd15141ca0" gracePeriod=30 Sep 16 07:54:45.697284 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.697243 2580 generic.go:356] "Generic (PLEG): container finished" podID="d2e6987a-0fa0-48ad-8101-9fc027c576e6" containerID="0d89814f794c47f70041111caa69d076d476ff2afd08ec9aacecbadd15141ca0" exitCode=0 Sep 16 07:54:45.697440 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.697292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" event={"ID":"d2e6987a-0fa0-48ad-8101-9fc027c576e6","Type":"ContainerDied","Data":"0d89814f794c47f70041111caa69d076d476ff2afd08ec9aacecbadd15141ca0"} Sep 16 07:54:45.734898 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.734836 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:54:45.847706 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.847673 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-metrics-certs\") pod \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " Sep 16 07:54:45.847853 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.847809 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-maas-api-tls\") pod \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " Sep 16 07:54:45.847853 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.847841 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e6987a-0fa0-48ad-8101-9fc027c576e6-openshift-service-ca\") pod \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " Sep 16 07:54:45.847946 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.847885 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z68f9\" (UniqueName: \"kubernetes.io/projected/d2e6987a-0fa0-48ad-8101-9fc027c576e6-kube-api-access-z68f9\") pod \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\" (UID: \"d2e6987a-0fa0-48ad-8101-9fc027c576e6\") " Sep 16 07:54:45.848179 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.848147 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e6987a-0fa0-48ad-8101-9fc027c576e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2e6987a-0fa0-48ad-8101-9fc027c576e6" (UID: "d2e6987a-0fa0-48ad-8101-9fc027c576e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:54:45.849827 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.849802 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d2e6987a-0fa0-48ad-8101-9fc027c576e6" (UID: "d2e6987a-0fa0-48ad-8101-9fc027c576e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:54:45.849975 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.849959 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2e6987a-0fa0-48ad-8101-9fc027c576e6" (UID: "d2e6987a-0fa0-48ad-8101-9fc027c576e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:54:45.850061 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.850042 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e6987a-0fa0-48ad-8101-9fc027c576e6-kube-api-access-z68f9" (OuterVolumeSpecName: "kube-api-access-z68f9") pod "d2e6987a-0fa0-48ad-8101-9fc027c576e6" (UID: "d2e6987a-0fa0-48ad-8101-9fc027c576e6"). InnerVolumeSpecName "kube-api-access-z68f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:54:45.948979 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.948946 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:45.948979 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.948973 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e6987a-0fa0-48ad-8101-9fc027c576e6-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:45.948979 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.948982 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68f9\" (UniqueName: \"kubernetes.io/projected/d2e6987a-0fa0-48ad-8101-9fc027c576e6-kube-api-access-z68f9\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:45.949190 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:45.948992 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e6987a-0fa0-48ad-8101-9fc027c576e6-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:54:46.705532 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:46.705506 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" Sep 16 07:54:46.705967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:46.705544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz" event={"ID":"d2e6987a-0fa0-48ad-8101-9fc027c576e6","Type":"ContainerDied","Data":"a37faea4f528b14f2ddf04d467415b2a90e4dcd29f28b12aeac34fe93b4228e7"} Sep 16 07:54:46.705967 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:46.705606 2580 scope.go:117] "RemoveContainer" containerID="0d89814f794c47f70041111caa69d076d476ff2afd08ec9aacecbadd15141ca0" Sep 16 07:54:46.735375 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:46.735350 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz"] Sep 16 07:54:46.738975 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:46.738953 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd359c-76b7758fff-jlwrz"] Sep 16 07:54:47.562727 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:47.562698 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e6987a-0fa0-48ad-8101-9fc027c576e6" path="/var/lib/kubelet/pods/d2e6987a-0fa0-48ad-8101-9fc027c576e6/volumes" Sep 16 07:54:58.297118 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:54:58.297073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:55:11.174658 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.174627 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9"] Sep 16 07:55:11.175067 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.175053 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b02c97c-ca40-4aed-9814-54e40eceadd2" containerName="main" Sep 16 07:55:11.175067 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.175067 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b02c97c-ca40-4aed-9814-54e40eceadd2" containerName="main" Sep 16 07:55:11.175142 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.175079 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92611524-dc02-4c2f-a048-ce8570aaada8" containerName="maas-api" Sep 16 07:55:11.175142 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.175085 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92611524-dc02-4c2f-a048-ce8570aaada8" containerName="maas-api" Sep 16 07:55:11.175142 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.175095 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e6987a-0fa0-48ad-8101-9fc027c576e6" containerName="maas-api" Sep 16 07:55:11.175142 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.175101 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e6987a-0fa0-48ad-8101-9fc027c576e6" containerName="maas-api" Sep 16 07:55:11.175322 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.175178 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92611524-dc02-4c2f-a048-ce8570aaada8" containerName="maas-api" Sep 16 07:55:11.175322 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.175188 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b02c97c-ca40-4aed-9814-54e40eceadd2" containerName="main" Sep 16 07:55:11.175322 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.175196 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e6987a-0fa0-48ad-8101-9fc027c576e6" containerName="maas-api" Sep 16 07:55:11.179579 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.179557 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.182189 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.182169 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fc2a294d\"/\"openshift-service-ca.crt\"" Sep 16 07:55:11.183556 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.183529 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fc2a294d\"/\"default-dockercfg-thlwr\"" Sep 16 07:55:11.183687 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.183561 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fc2a294d\"/\"kube-root-ca.crt\"" Sep 16 07:55:11.183687 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.183543 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fc2a294d\"/\"e2e-lifecycle-model-fc2a294d-kserve-self-signed-certs\"" Sep 16 07:55:11.187854 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.187834 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9"] Sep 16 07:55:11.279599 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.279565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-model-cache\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.279762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.279640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-home\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.279762 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.279694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5vp2\" (UniqueName: \"kubernetes.io/projected/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-kube-api-access-x5vp2\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.279880 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.279760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-dshm\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.279880 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.279811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tls-certs\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.279992 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.279950 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tmp-dir\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.380722 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.380693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tmp-dir\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.380852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.380729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-model-cache\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.380852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.380754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-home\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.380852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.380778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5vp2\" (UniqueName: \"kubernetes.io/projected/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-kube-api-access-x5vp2\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.380852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.380808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-dshm\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.380852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.380827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tls-certs\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.381165 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.381135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-model-cache\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.381257 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.381161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-home\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.381257 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.381182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tmp-dir\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.383038 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.383011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-dshm\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.383298 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.383281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tls-certs\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.389077 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.389046 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5vp2\" (UniqueName: \"kubernetes.io/projected/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-kube-api-access-x5vp2\") pod \"e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.490249 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.490170 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:11.761641 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.761618 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9"] Sep 16 07:55:11.763371 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:55:11.763342 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda07f5b8e_67b0_4183_82c5_e38a9d8188dd.slice/crio-b7ec5be7dbd516a279b5255276922345f83dcc3f48cc084655e3dd43e1cc2581 WatchSource:0}: Error finding container b7ec5be7dbd516a279b5255276922345f83dcc3f48cc084655e3dd43e1cc2581: Status 404 returned error can't find the container with id b7ec5be7dbd516a279b5255276922345f83dcc3f48cc084655e3dd43e1cc2581 Sep 16 07:55:11.849120 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.849093 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" event={"ID":"a07f5b8e-67b0-4183-82c5-e38a9d8188dd","Type":"ContainerStarted","Data":"c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613"} Sep 16 07:55:11.849241 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.849137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" event={"ID":"a07f5b8e-67b0-4183-82c5-e38a9d8188dd","Type":"ContainerStarted","Data":"b7ec5be7dbd516a279b5255276922345f83dcc3f48cc084655e3dd43e1cc2581"} Sep 16 07:55:11.866005 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:11.865960 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" podStartSLOduration=0.865943864 podStartE2EDuration="865.943864ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:55:11.86494726 +0000 UTC m=+1284.784563211" watchObservedRunningTime="2026-09-16 07:55:11.865943864 +0000 UTC m=+1284.785559809" Sep 16 07:55:17.401798 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:17.401763 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:55:21.491387 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:21.491345 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:21.491791 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:21.491407 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:21.504133 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:21.504107 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:21.919936 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:21.919911 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:55:42.037640 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.037607 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2"] Sep 16 07:55:42.041536 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.041515 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.044390 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.044360 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-3616cc-serving-cert\"" Sep 16 07:55:42.052009 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.051985 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2"] Sep 16 07:55:42.142589 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.142562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kx68\" (UniqueName: \"kubernetes.io/projected/ffc8796c-0eef-4ea7-b618-270cb14a6401-kube-api-access-2kx68\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.142729 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.142611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc8796c-0eef-4ea7-b618-270cb14a6401-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.142729 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.142676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.142805 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.142738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.243508 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.243481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc8796c-0eef-4ea7-b618-270cb14a6401-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.243656 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.243522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.243656 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.243556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.243656 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.243591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kx68\" (UniqueName: \"kubernetes.io/projected/ffc8796c-0eef-4ea7-b618-270cb14a6401-kube-api-access-2kx68\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.244197 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.244164 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc8796c-0eef-4ea7-b618-270cb14a6401-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.246055 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.246029 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.246167 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.246116 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.251091 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.251068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kx68\" (UniqueName: \"kubernetes.io/projected/ffc8796c-0eef-4ea7-b618-270cb14a6401-kube-api-access-2kx68\") pod \"maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.356433 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.356362 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:42.645794 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:55:42.645761 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc8796c_0eef_4ea7_b618_270cb14a6401.slice/crio-27697abe27fdd566a55548573ebdf6022316822c384e83627430b4401590899e WatchSource:0}: Error finding container 27697abe27fdd566a55548573ebdf6022316822c384e83627430b4401590899e: Status 404 returned error can't find the container with id 27697abe27fdd566a55548573ebdf6022316822c384e83627430b4401590899e Sep 16 07:55:42.647324 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.647299 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:55:42.648057 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:42.648036 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2"] Sep 16 07:55:43.041520 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:43.041490 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" event={"ID":"ffc8796c-0eef-4ea7-b618-270cb14a6401","Type":"ContainerStarted","Data":"27697abe27fdd566a55548573ebdf6022316822c384e83627430b4401590899e"} Sep 16 07:55:44.049876 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:44.049842 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" event={"ID":"ffc8796c-0eef-4ea7-b618-270cb14a6401","Type":"ContainerStarted","Data":"fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e"} Sep 16 07:55:44.050279 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:44.049902 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:44.068389 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:44.068341 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" podStartSLOduration=1.711843631 podStartE2EDuration="2.068326414s" totalImagesPullingTime="356.482783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:55:44.066419454 +0000 UTC m=+1316.986035405" watchObservedRunningTime="2026-09-16 07:55:44.068326414 +0000 UTC m=+1316.987942360" Sep 16 07:55:50.061055 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:50.061021 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 07:55:55.903661 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:55:55.903628 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:56:14.139442 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.139348 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t"] Sep 16 07:56:14.139981 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.139629 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" podUID="904d5328-61af-45a7-8fc5-ef936932733a" containerName="maas-api" containerID="cri-o://30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2" gracePeriod=30 Sep 16 07:56:14.397568 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.397507 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:56:14.523132 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.523096 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-metrics-certs\") pod \"904d5328-61af-45a7-8fc5-ef936932733a\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " Sep 16 07:56:14.523343 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.523166 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc24c\" (UniqueName: \"kubernetes.io/projected/904d5328-61af-45a7-8fc5-ef936932733a-kube-api-access-cc24c\") pod \"904d5328-61af-45a7-8fc5-ef936932733a\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " Sep 16 07:56:14.523343 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.523281 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904d5328-61af-45a7-8fc5-ef936932733a-openshift-service-ca\") pod \"904d5328-61af-45a7-8fc5-ef936932733a\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " Sep 16 07:56:14.523477 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.523373 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-maas-api-tls\") pod \"904d5328-61af-45a7-8fc5-ef936932733a\" (UID: \"904d5328-61af-45a7-8fc5-ef936932733a\") " Sep 16 07:56:14.523665 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.523625 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904d5328-61af-45a7-8fc5-ef936932733a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "904d5328-61af-45a7-8fc5-ef936932733a" (UID: "904d5328-61af-45a7-8fc5-ef936932733a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:56:14.523812 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.523768 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904d5328-61af-45a7-8fc5-ef936932733a-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:56:14.528802 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.528690 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "904d5328-61af-45a7-8fc5-ef936932733a" (UID: "904d5328-61af-45a7-8fc5-ef936932733a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:56:14.528802 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.528707 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904d5328-61af-45a7-8fc5-ef936932733a-kube-api-access-cc24c" (OuterVolumeSpecName: "kube-api-access-cc24c") pod "904d5328-61af-45a7-8fc5-ef936932733a" (UID: "904d5328-61af-45a7-8fc5-ef936932733a"). InnerVolumeSpecName "kube-api-access-cc24c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:56:14.528802 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.528774 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "904d5328-61af-45a7-8fc5-ef936932733a" (UID: "904d5328-61af-45a7-8fc5-ef936932733a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:56:14.624748 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.624714 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:56:14.624748 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.624744 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc24c\" (UniqueName: \"kubernetes.io/projected/904d5328-61af-45a7-8fc5-ef936932733a-kube-api-access-cc24c\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:56:14.624748 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:14.624756 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d5328-61af-45a7-8fc5-ef936932733a-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:56:15.241506 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:15.241367 2580 generic.go:356] "Generic (PLEG): container finished" podID="904d5328-61af-45a7-8fc5-ef936932733a" containerID="30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2" exitCode=0 Sep 16 07:56:15.241506 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:15.241421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" event={"ID":"904d5328-61af-45a7-8fc5-ef936932733a","Type":"ContainerDied","Data":"30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2"} Sep 16 07:56:15.241506 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:15.241447 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" Sep 16 07:56:15.241506 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:15.241505 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t" event={"ID":"904d5328-61af-45a7-8fc5-ef936932733a","Type":"ContainerDied","Data":"0e34915128e01c7367622ec0ba8ea19d6bcb4a74baeeb0b937e73333377b1de1"} Sep 16 07:56:15.242139 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:15.241558 2580 scope.go:117] "RemoveContainer" containerID="30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2" Sep 16 07:56:15.251995 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:15.251975 2580 scope.go:117] "RemoveContainer" containerID="30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2" Sep 16 07:56:15.252310 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:56:15.252290 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2\": container with ID starting with 30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2 not found: ID does not exist" containerID="30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2" Sep 16 07:56:15.252392 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:15.252317 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2"} err="failed to get container status \"30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2\": rpc error: code = NotFound desc = could not find container \"30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2\": container with ID starting with 30bb54c266d01ab23e0d99ec1302d147cbb0828ff1b7d82072975d1a795a42b2 not found: ID does not exist" Sep 16 07:56:15.265282 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:15.265261 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t"] Sep 16 07:56:15.269185 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:15.269163 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc2a294d-6478f74f8c-8jr6t"] Sep 16 07:56:15.560951 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:15.560882 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="904d5328-61af-45a7-8fc5-ef936932733a" path="/var/lib/kubelet/pods/904d5328-61af-45a7-8fc5-ef936932733a/volumes" Sep 16 07:56:16.687885 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.687850 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n"] Sep 16 07:56:16.688496 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.688474 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904d5328-61af-45a7-8fc5-ef936932733a" containerName="maas-api" Sep 16 07:56:16.688552 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.688499 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="904d5328-61af-45a7-8fc5-ef936932733a" containerName="maas-api" Sep 16 07:56:16.688620 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.688608 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="904d5328-61af-45a7-8fc5-ef936932733a" containerName="maas-api" Sep 16 07:56:16.693498 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.693473 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.696225 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.696194 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e9fc8c-serving-cert\"" Sep 16 07:56:16.701769 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.701603 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n"] Sep 16 07:56:16.844429 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.844396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.844579 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.844440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.844579 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.844521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.844660 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.844596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmls6\" (UniqueName: \"kubernetes.io/projected/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-kube-api-access-rmls6\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.945598 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.945522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.945598 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.945565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.945756 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.945621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.945756 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.945694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmls6\" (UniqueName: \"kubernetes.io/projected/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-kube-api-access-rmls6\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.945841 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:56:16.945820 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-serving-cert: secret "maas-api-e2e-shared-b-w3-e9fc8c-serving-cert" not found Sep 16 07:56:16.945894 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:56:16.945884 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-maas-api-tls podName:9f301562-0f6f-4c51-bfce-bf0a2e8b405c nodeName:}" failed. No retries permitted until 2026-09-16 07:56:17.445867805 +0000 UTC m=+1350.365483729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-maas-api-tls") pod "maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" (UID: "9f301562-0f6f-4c51-bfce-bf0a2e8b405c") : secret "maas-api-e2e-shared-b-w3-e9fc8c-serving-cert" not found Sep 16 07:56:16.946269 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.946245 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.948166 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.948137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:16.954007 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:16.953984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmls6\" (UniqueName: \"kubernetes.io/projected/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-kube-api-access-rmls6\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:17.449931 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:17.449897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:17.452428 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:17.452395 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:17.605481 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:17.605452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:17.886652 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:17.886629 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n"] Sep 16 07:56:17.887616 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:56:17.887590 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f301562_0f6f_4c51_bfce_bf0a2e8b405c.slice/crio-5760ff2f2510f246e0c2ca4cacbaaf84cf681668479f0fce16404ec9f49f0fa7 WatchSource:0}: Error finding container 5760ff2f2510f246e0c2ca4cacbaaf84cf681668479f0fce16404ec9f49f0fa7: Status 404 returned error can't find the container with id 5760ff2f2510f246e0c2ca4cacbaaf84cf681668479f0fce16404ec9f49f0fa7 Sep 16 07:56:18.263110 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:18.263071 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" event={"ID":"9f301562-0f6f-4c51-bfce-bf0a2e8b405c","Type":"ContainerStarted","Data":"5760ff2f2510f246e0c2ca4cacbaaf84cf681668479f0fce16404ec9f49f0fa7"} Sep 16 07:56:19.271174 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:19.271135 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" event={"ID":"9f301562-0f6f-4c51-bfce-bf0a2e8b405c","Type":"ContainerStarted","Data":"1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3"} Sep 16 07:56:19.271615 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:19.271202 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:19.293500 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:19.293441 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" podStartSLOduration=2.7987130479999998 podStartE2EDuration="3.293426234s" totalImagesPullingTime="494.713186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:56:19.291883187 +0000 UTC m=+1352.211499134" watchObservedRunningTime="2026-09-16 07:56:19.293426234 +0000 UTC m=+1352.213042202" Sep 16 07:56:22.844583 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:22.844547 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9"] Sep 16 07:56:25.282743 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:25.282712 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 07:56:35.808334 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:35.808299 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:56:37.864470 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:37.864430 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" podUID="a07f5b8e-67b0-4183-82c5-e38a9d8188dd" containerName="main" containerID="cri-o://c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613" gracePeriod=45 Sep 16 07:56:38.118535 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.118475 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:56:38.218728 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.218686 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-model-cache\") pod \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " Sep 16 07:56:38.218888 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.218744 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tls-certs\") pod \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " Sep 16 07:56:38.218888 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.218779 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tmp-dir\") pod \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " Sep 16 07:56:38.218888 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.218848 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5vp2\" (UniqueName: \"kubernetes.io/projected/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-kube-api-access-x5vp2\") pod \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " Sep 16 07:56:38.218888 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.218877 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-dshm\") pod \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " Sep 16 07:56:38.219085 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.218931 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "a07f5b8e-67b0-4183-82c5-e38a9d8188dd" (UID: "a07f5b8e-67b0-4183-82c5-e38a9d8188dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:56:38.219085 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.218946 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-home\") pod \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\" (UID: \"a07f5b8e-67b0-4183-82c5-e38a9d8188dd\") " Sep 16 07:56:38.219085 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.219066 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a07f5b8e-67b0-4183-82c5-e38a9d8188dd" (UID: "a07f5b8e-67b0-4183-82c5-e38a9d8188dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:56:38.219270 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.219148 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-home" (OuterVolumeSpecName: "home") pod "a07f5b8e-67b0-4183-82c5-e38a9d8188dd" (UID: "a07f5b8e-67b0-4183-82c5-e38a9d8188dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:56:38.219361 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.219342 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-model-cache\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:56:38.219412 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.219370 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tmp-dir\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:56:38.219412 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.219388 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-home\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:56:38.221037 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.221004 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-dshm" (OuterVolumeSpecName: "dshm") pod "a07f5b8e-67b0-4183-82c5-e38a9d8188dd" (UID: "a07f5b8e-67b0-4183-82c5-e38a9d8188dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:56:38.221160 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.221034 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-kube-api-access-x5vp2" (OuterVolumeSpecName: "kube-api-access-x5vp2") pod "a07f5b8e-67b0-4183-82c5-e38a9d8188dd" (UID: "a07f5b8e-67b0-4183-82c5-e38a9d8188dd"). InnerVolumeSpecName "kube-api-access-x5vp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:56:38.221160 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.221090 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a07f5b8e-67b0-4183-82c5-e38a9d8188dd" (UID: "a07f5b8e-67b0-4183-82c5-e38a9d8188dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:56:38.320049 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.320021 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5vp2\" (UniqueName: \"kubernetes.io/projected/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-kube-api-access-x5vp2\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:56:38.320049 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.320044 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-dshm\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:56:38.320049 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.320055 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07f5b8e-67b0-4183-82c5-e38a9d8188dd-tls-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:56:38.385055 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.384982 2580 generic.go:356] "Generic (PLEG): container finished" podID="a07f5b8e-67b0-4183-82c5-e38a9d8188dd" containerID="c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613" exitCode=0 Sep 16 07:56:38.385055 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.385030 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" event={"ID":"a07f5b8e-67b0-4183-82c5-e38a9d8188dd","Type":"ContainerDied","Data":"c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613"} Sep 16 07:56:38.385278 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.385061 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" event={"ID":"a07f5b8e-67b0-4183-82c5-e38a9d8188dd","Type":"ContainerDied","Data":"b7ec5be7dbd516a279b5255276922345f83dcc3f48cc084655e3dd43e1cc2581"} Sep 16 07:56:38.385278 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.385063 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9" Sep 16 07:56:38.385278 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.385078 2580 scope.go:117] "RemoveContainer" containerID="c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613" Sep 16 07:56:38.396261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.396241 2580 scope.go:117] "RemoveContainer" containerID="c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613" Sep 16 07:56:38.396548 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:56:38.396529 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613\": container with ID starting with c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613 not found: ID does not exist" containerID="c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613" Sep 16 07:56:38.396602 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.396557 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613"} err="failed to get container status \"c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613\": rpc error: code = NotFound desc = could not find container \"c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613\": container with ID starting with c245aff59ac2f00cf897315f5d0423721ca461bb9dc9b4fbfa3d2c5f47ceb613 not found: ID does not exist" Sep 16 07:56:38.408765 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.408740 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9"] Sep 16 07:56:38.414396 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:38.414375 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fc2a294d/e2e-lifecycle-model-fc2a294d-kserve-6b7b5ccfd7-bntt9"] Sep 16 07:56:39.560803 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:39.560756 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a07f5b8e-67b0-4183-82c5-e38a9d8188dd" path="/var/lib/kubelet/pods/a07f5b8e-67b0-4183-82c5-e38a9d8188dd/volumes" Sep 16 07:56:48.870751 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.870715 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv"] Sep 16 07:56:48.871480 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.871459 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a07f5b8e-67b0-4183-82c5-e38a9d8188dd" containerName="main" Sep 16 07:56:48.871546 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.871485 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a07f5b8e-67b0-4183-82c5-e38a9d8188dd" containerName="main" Sep 16 07:56:48.871644 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.871631 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a07f5b8e-67b0-4183-82c5-e38a9d8188dd" containerName="main" Sep 16 07:56:48.876374 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.876351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:48.879704 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.879533 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"distinct-w4-fc27a5-kserve-self-signed-certs\"" Sep 16 07:56:48.879704 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.879557 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"kube-root-ca.crt\"" Sep 16 07:56:48.879882 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.879794 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"default-dockercfg-dkpl6\"" Sep 16 07:56:48.879993 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.879971 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"openshift-service-ca.crt\"" Sep 16 07:56:48.882056 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.882034 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv"] Sep 16 07:56:48.909137 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.909112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-home\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:48.909279 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.909149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f8dd92-eb73-43c8-aa56-78988fc09b10-tls-certs\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:48.909334 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.909273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2nt\" (UniqueName: \"kubernetes.io/projected/56f8dd92-eb73-43c8-aa56-78988fc09b10-kube-api-access-2k2nt\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:48.909334 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.909308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-tmp-dir\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:48.909426 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.909349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-model-cache\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:48.909426 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:48.909377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-dshm\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.010106 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.010078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2nt\" (UniqueName: \"kubernetes.io/projected/56f8dd92-eb73-43c8-aa56-78988fc09b10-kube-api-access-2k2nt\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.010261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.010114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-tmp-dir\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.010261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.010152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-model-cache\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.010261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.010187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-dshm\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.010261 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.010253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-home\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.010462 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.010286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f8dd92-eb73-43c8-aa56-78988fc09b10-tls-certs\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.010594 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.010568 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-tmp-dir\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.010659 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.010582 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-model-cache\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.010720 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.010674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-home\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.012509 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.012483 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-dshm\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.012882 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.012862 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f8dd92-eb73-43c8-aa56-78988fc09b10-tls-certs\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.017851 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.017831 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2nt\" (UniqueName: \"kubernetes.io/projected/56f8dd92-eb73-43c8-aa56-78988fc09b10-kube-api-access-2k2nt\") pod \"distinct-w4-fc27a5-kserve-84c67f9985-t4lfv\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.188879 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.188852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:49.468833 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:49.468800 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv"] Sep 16 07:56:49.469342 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:56:49.469316 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56f8dd92_eb73_43c8_aa56_78988fc09b10.slice/crio-3b02819c4af59e36224a6325c607f3f32ceae2382db22eb2b30fb999dae9071d WatchSource:0}: Error finding container 3b02819c4af59e36224a6325c607f3f32ceae2382db22eb2b30fb999dae9071d: Status 404 returned error can't find the container with id 3b02819c4af59e36224a6325c607f3f32ceae2382db22eb2b30fb999dae9071d Sep 16 07:56:50.467310 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:50.467266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" event={"ID":"56f8dd92-eb73-43c8-aa56-78988fc09b10","Type":"ContainerStarted","Data":"919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556"} Sep 16 07:56:50.467670 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:50.467320 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" event={"ID":"56f8dd92-eb73-43c8-aa56-78988fc09b10","Type":"ContainerStarted","Data":"3b02819c4af59e36224a6325c607f3f32ceae2382db22eb2b30fb999dae9071d"} Sep 16 07:56:50.486435 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:50.486382 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" podStartSLOduration=2.486367791 podStartE2EDuration="2.486367791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:56:50.484260781 +0000 UTC m=+1383.403876728" watchObservedRunningTime="2026-09-16 07:56:50.486367791 +0000 UTC m=+1383.405983729" Sep 16 07:56:59.189016 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:59.188975 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:59.189421 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:59.189028 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:59.201227 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:59.201183 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:56:59.535328 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:56:59.535253 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 07:57:00.503809 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:57:00.503773 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:57:33.098286 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:57:33.098249 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:58:03.807072 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:03.807033 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:58:27.803884 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:27.803847 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:58:27.965898 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:27.965868 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5"] Sep 16 07:58:27.969663 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:27.969640 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:27.972359 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:27.972337 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"embedding-w0-c9020d-kserve-self-signed-certs\"" Sep 16 07:58:27.979540 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:27.979517 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5"] Sep 16 07:58:28.094050 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.093977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-model-cache\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.094050 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.094027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95qhw\" (UniqueName: \"kubernetes.io/projected/969d293b-61ed-4e14-b7fa-0fc9a2987afb-kube-api-access-95qhw\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.094236 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.094073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tls-certs\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.094236 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.094100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-home\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.094236 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.094132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-dshm\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.094236 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.094204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tmp-dir\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.194639 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.194609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tmp-dir\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.194771 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.194659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-model-cache\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.194771 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.194686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qhw\" (UniqueName: \"kubernetes.io/projected/969d293b-61ed-4e14-b7fa-0fc9a2987afb-kube-api-access-95qhw\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.194771 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.194706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tls-certs\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.194771 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.194728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-home\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.194771 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.194753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-dshm\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.195075 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.195054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tmp-dir\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.195137 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.195100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-model-cache\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.195196 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.195129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-home\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.196986 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.196966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-dshm\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.197383 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.197365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tls-certs\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.202450 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.202430 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qhw\" (UniqueName: \"kubernetes.io/projected/969d293b-61ed-4e14-b7fa-0fc9a2987afb-kube-api-access-95qhw\") pod \"embedding-w0-c9020d-kserve-75769ddc46-zf9t5\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.279637 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.279613 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:28.555941 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:28.555913 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5"] Sep 16 07:58:28.557648 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:58:28.557617 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod969d293b_61ed_4e14_b7fa_0fc9a2987afb.slice/crio-a2e9ffa45a02fcb7a0b6e8e55ecb1fc07c06c8db20309bbcb7b2ab58e44a13c2 WatchSource:0}: Error finding container a2e9ffa45a02fcb7a0b6e8e55ecb1fc07c06c8db20309bbcb7b2ab58e44a13c2: Status 404 returned error can't find the container with id a2e9ffa45a02fcb7a0b6e8e55ecb1fc07c06c8db20309bbcb7b2ab58e44a13c2 Sep 16 07:58:29.077750 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:29.077714 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" event={"ID":"969d293b-61ed-4e14-b7fa-0fc9a2987afb","Type":"ContainerStarted","Data":"165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0"} Sep 16 07:58:29.077750 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:29.077755 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" event={"ID":"969d293b-61ed-4e14-b7fa-0fc9a2987afb","Type":"ContainerStarted","Data":"a2e9ffa45a02fcb7a0b6e8e55ecb1fc07c06c8db20309bbcb7b2ab58e44a13c2"} Sep 16 07:58:29.101047 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:29.100999 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" podStartSLOduration=2.100982854 podStartE2EDuration="2.100982854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:58:29.098410274 +0000 UTC m=+1482.018026241" watchObservedRunningTime="2026-09-16 07:58:29.100982854 +0000 UTC m=+1482.020598800" Sep 16 07:58:38.279986 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.279950 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:38.280622 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.280006 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:38.292905 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.292878 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:38.858340 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.858305 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x"] Sep 16 07:58:38.863186 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.863162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:38.865766 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.865739 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-987ea6-serving-cert\"" Sep 16 07:58:38.870395 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.870362 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x"] Sep 16 07:58:38.995276 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.995244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4edb58f-4711-4031-aaee-796423e3567c-openshift-service-ca\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:38.995454 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.995310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-maas-api-tls\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:38.995454 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.995330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-metrics-certs\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:38.995454 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:38.995434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kr25\" (UniqueName: \"kubernetes.io/projected/e4edb58f-4711-4031-aaee-796423e3567c-kube-api-access-9kr25\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:39.096932 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.096902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-maas-api-tls\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:39.096932 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.096934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-metrics-certs\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:39.097135 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.096972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kr25\" (UniqueName: \"kubernetes.io/projected/e4edb58f-4711-4031-aaee-796423e3567c-kube-api-access-9kr25\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:39.097135 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.097029 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4edb58f-4711-4031-aaee-796423e3567c-openshift-service-ca\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:39.097708 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.097683 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4edb58f-4711-4031-aaee-796423e3567c-openshift-service-ca\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:39.099382 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.099357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-metrics-certs\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:39.099514 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.099492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-maas-api-tls\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:39.104949 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.104926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kr25\" (UniqueName: \"kubernetes.io/projected/e4edb58f-4711-4031-aaee-796423e3567c-kube-api-access-9kr25\") pod \"maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:39.151416 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.151356 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 07:58:39.176541 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.176520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:39.459698 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:39.459667 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x"] Sep 16 07:58:39.461161 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:58:39.461134 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4edb58f_4711_4031_aaee_796423e3567c.slice/crio-66d8ea2b4a43f4d4e76cd6416e9ea1ec2b8586ff2c74007a6f8030a82010dfe4 WatchSource:0}: Error finding container 66d8ea2b4a43f4d4e76cd6416e9ea1ec2b8586ff2c74007a6f8030a82010dfe4: Status 404 returned error can't find the container with id 66d8ea2b4a43f4d4e76cd6416e9ea1ec2b8586ff2c74007a6f8030a82010dfe4 Sep 16 07:58:40.154434 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:40.154393 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" event={"ID":"e4edb58f-4711-4031-aaee-796423e3567c","Type":"ContainerStarted","Data":"2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a"} Sep 16 07:58:40.154591 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:40.154485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" event={"ID":"e4edb58f-4711-4031-aaee-796423e3567c","Type":"ContainerStarted","Data":"66d8ea2b4a43f4d4e76cd6416e9ea1ec2b8586ff2c74007a6f8030a82010dfe4"} Sep 16 07:58:40.173852 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:40.173805 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" podStartSLOduration=1.75101473 podStartE2EDuration="2.173791966s" totalImagesPullingTime="422.777236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:58:40.171003559 +0000 UTC m=+1493.090619517" watchObservedRunningTime="2026-09-16 07:58:40.173791966 +0000 UTC m=+1493.093407913" Sep 16 07:58:41.160429 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:41.160399 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:47.172023 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:47.171990 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:58:47.594185 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:47.594117 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:58:47.598941 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:58:47.598918 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 07:59:07.208523 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:07.208490 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 07:59:13.769020 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.768941 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps"] Sep 16 07:59:13.776397 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.776372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.779496 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.779471 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3a5b4c-serving-cert\"" Sep 16 07:59:13.782325 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.782299 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps"] Sep 16 07:59:13.890605 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.890573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-maas-api-tls\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.890782 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.890703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d2c21c-4653-4a06-9c8f-c3b816f76747-openshift-service-ca\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.890782 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.890734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htzjh\" (UniqueName: \"kubernetes.io/projected/63d2c21c-4653-4a06-9c8f-c3b816f76747-kube-api-access-htzjh\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.890782 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.890763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-metrics-certs\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.992039 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.992002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d2c21c-4653-4a06-9c8f-c3b816f76747-openshift-service-ca\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.992288 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.992045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htzjh\" (UniqueName: \"kubernetes.io/projected/63d2c21c-4653-4a06-9c8f-c3b816f76747-kube-api-access-htzjh\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.992288 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.992173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-metrics-certs\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.992429 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.992290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-maas-api-tls\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.992757 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.992728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d2c21c-4653-4a06-9c8f-c3b816f76747-openshift-service-ca\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.994598 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.994578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-metrics-certs\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:13.994745 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:13.994724 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-maas-api-tls\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:14.002385 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:14.002366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htzjh\" (UniqueName: \"kubernetes.io/projected/63d2c21c-4653-4a06-9c8f-c3b816f76747-kube-api-access-htzjh\") pod \"maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:14.090364 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:14.090275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:14.368591 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:14.368567 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps"] Sep 16 07:59:14.370645 ip-10-0-131-134 kubenswrapper[2580]: W0916 07:59:14.370618 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d2c21c_4653_4a06_9c8f_c3b816f76747.slice/crio-11298b5d977ac77734a079d87ca06de2e13ee101ac57ce5aae09c2c61a7dacfc WatchSource:0}: Error finding container 11298b5d977ac77734a079d87ca06de2e13ee101ac57ce5aae09c2c61a7dacfc: Status 404 returned error can't find the container with id 11298b5d977ac77734a079d87ca06de2e13ee101ac57ce5aae09c2c61a7dacfc Sep 16 07:59:15.372931 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:15.372890 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" event={"ID":"63d2c21c-4653-4a06-9c8f-c3b816f76747","Type":"ContainerStarted","Data":"9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780"} Sep 16 07:59:15.373325 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:15.372967 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:15.373325 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:15.372997 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" event={"ID":"63d2c21c-4653-4a06-9c8f-c3b816f76747","Type":"ContainerStarted","Data":"11298b5d977ac77734a079d87ca06de2e13ee101ac57ce5aae09c2c61a7dacfc"} Sep 16 07:59:15.391660 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:15.391608 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" podStartSLOduration=2.057332084 podStartE2EDuration="2.391593312s" totalImagesPullingTime="334.261228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:59:15.389165475 +0000 UTC m=+1528.308781650" watchObservedRunningTime="2026-09-16 07:59:15.391593312 +0000 UTC m=+1528.311209257" Sep 16 07:59:21.384310 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:21.384276 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 07:59:44.748516 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:44.748481 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s"] Sep 16 07:59:44.759836 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:44.759805 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5"] Sep 16 07:59:55.838557 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:55.838521 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697"] Sep 16 07:59:55.839021 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:55.838746 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" podUID="4557169e-72cc-4a7d-b42f-e87a5c1608a3" containerName="maas-api" containerID="cri-o://d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd" gracePeriod=30 Sep 16 07:59:55.891058 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:55.891019 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2"] Sep 16 07:59:55.891328 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:55.891294 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" podUID="79178c36-2949-484f-b547-aed1a748fd50" containerName="payload-processing" containerID="cri-o://ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4" gracePeriod=30 Sep 16 07:59:56.190491 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.190461 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:59:56.278810 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.278779 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nvtl\" (UniqueName: \"kubernetes.io/projected/79178c36-2949-484f-b547-aed1a748fd50-kube-api-access-5nvtl\") pod \"79178c36-2949-484f-b547-aed1a748fd50\" (UID: \"79178c36-2949-484f-b547-aed1a748fd50\") " Sep 16 07:59:56.278958 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.278896 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79178c36-2949-484f-b547-aed1a748fd50-plugins-config-volume\") pod \"79178c36-2949-484f-b547-aed1a748fd50\" (UID: \"79178c36-2949-484f-b547-aed1a748fd50\") " Sep 16 07:59:56.279376 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.279342 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79178c36-2949-484f-b547-aed1a748fd50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79178c36-2949-484f-b547-aed1a748fd50" (UID: "79178c36-2949-484f-b547-aed1a748fd50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:59:56.280815 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.280780 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79178c36-2949-484f-b547-aed1a748fd50-kube-api-access-5nvtl" (OuterVolumeSpecName: "kube-api-access-5nvtl") pod "79178c36-2949-484f-b547-aed1a748fd50" (UID: "79178c36-2949-484f-b547-aed1a748fd50"). InnerVolumeSpecName "kube-api-access-5nvtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:59:56.306496 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.306473 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:59:56.379948 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.379864 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-metrics-certs\") pod \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " Sep 16 07:59:56.379948 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.379933 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqds5\" (UniqueName: \"kubernetes.io/projected/4557169e-72cc-4a7d-b42f-e87a5c1608a3-kube-api-access-kqds5\") pod \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " Sep 16 07:59:56.380158 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.380016 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4557169e-72cc-4a7d-b42f-e87a5c1608a3-openshift-service-ca\") pod \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " Sep 16 07:59:56.380158 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.380049 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-maas-api-tls\") pod \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\" (UID: \"4557169e-72cc-4a7d-b42f-e87a5c1608a3\") " Sep 16 07:59:56.380428 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.380384 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79178c36-2949-484f-b547-aed1a748fd50-plugins-config-volume\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:59:56.380428 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.380411 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nvtl\" (UniqueName: \"kubernetes.io/projected/79178c36-2949-484f-b547-aed1a748fd50-kube-api-access-5nvtl\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:59:56.380709 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.380668 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4557169e-72cc-4a7d-b42f-e87a5c1608a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4557169e-72cc-4a7d-b42f-e87a5c1608a3" (UID: "4557169e-72cc-4a7d-b42f-e87a5c1608a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:59:56.382521 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.382478 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4557169e-72cc-4a7d-b42f-e87a5c1608a3" (UID: "4557169e-72cc-4a7d-b42f-e87a5c1608a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:59:56.382847 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.382817 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4557169e-72cc-4a7d-b42f-e87a5c1608a3-kube-api-access-kqds5" (OuterVolumeSpecName: "kube-api-access-kqds5") pod "4557169e-72cc-4a7d-b42f-e87a5c1608a3" (UID: "4557169e-72cc-4a7d-b42f-e87a5c1608a3"). InnerVolumeSpecName "kube-api-access-kqds5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:59:56.382970 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.382914 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4557169e-72cc-4a7d-b42f-e87a5c1608a3" (UID: "4557169e-72cc-4a7d-b42f-e87a5c1608a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:59:56.480878 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.480848 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4557169e-72cc-4a7d-b42f-e87a5c1608a3-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:59:56.480878 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.480873 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:59:56.480878 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.480882 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4557169e-72cc-4a7d-b42f-e87a5c1608a3-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:59:56.481093 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.480890 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqds5\" (UniqueName: \"kubernetes.io/projected/4557169e-72cc-4a7d-b42f-e87a5c1608a3-kube-api-access-kqds5\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:59:56.625446 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.625410 2580 generic.go:356] "Generic (PLEG): container finished" podID="79178c36-2949-484f-b547-aed1a748fd50" containerID="ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4" exitCode=0 Sep 16 07:59:56.625573 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.625451 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" Sep 16 07:59:56.625573 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.625498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" event={"ID":"79178c36-2949-484f-b547-aed1a748fd50","Type":"ContainerDied","Data":"ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4"} Sep 16 07:59:56.625573 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.625545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2" event={"ID":"79178c36-2949-484f-b547-aed1a748fd50","Type":"ContainerDied","Data":"19b83d247ce16f4cce107e6c3f4f617d6a9a86ac1e56981376c9ea45e7ffb64f"} Sep 16 07:59:56.625573 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.625571 2580 scope.go:117] "RemoveContainer" containerID="ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4" Sep 16 07:59:56.627145 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.627123 2580 generic.go:356] "Generic (PLEG): container finished" podID="4557169e-72cc-4a7d-b42f-e87a5c1608a3" containerID="d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd" exitCode=0 Sep 16 07:59:56.627269 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.627163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" event={"ID":"4557169e-72cc-4a7d-b42f-e87a5c1608a3","Type":"ContainerDied","Data":"d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd"} Sep 16 07:59:56.627269 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.627177 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" Sep 16 07:59:56.627269 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.627200 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697" event={"ID":"4557169e-72cc-4a7d-b42f-e87a5c1608a3","Type":"ContainerDied","Data":"9644efe183df8bb85df5efa5d9818804a53084eb4fd9750dd521eb4a2406d6a2"} Sep 16 07:59:56.635487 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.635468 2580 scope.go:117] "RemoveContainer" containerID="ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4" Sep 16 07:59:56.635761 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:59:56.635733 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4\": container with ID starting with ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4 not found: ID does not exist" containerID="ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4" Sep 16 07:59:56.635856 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.635773 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4"} err="failed to get container status \"ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4\": rpc error: code = NotFound desc = could not find container \"ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4\": container with ID starting with ff4a5fefff5cdd6e1bdbd05480ab91a6855eeb6bed2a6a99004ddcacb30d58b4 not found: ID does not exist" Sep 16 07:59:56.635856 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.635796 2580 scope.go:117] "RemoveContainer" containerID="d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd" Sep 16 07:59:56.644682 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.644665 2580 scope.go:117] "RemoveContainer" containerID="d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd" Sep 16 07:59:56.644943 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:59:56.644922 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd\": container with ID starting with d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd not found: ID does not exist" containerID="d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd" Sep 16 07:59:56.644986 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.644949 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd"} err="failed to get container status \"d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd\": rpc error: code = NotFound desc = could not find container \"d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd\": container with ID starting with d6905289acd88fd77b1022434a56decf0bf2402c148d85f26115a4985384cefd not found: ID does not exist" Sep 16 07:59:56.652242 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.652201 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697"] Sep 16 07:59:56.656127 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.656106 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c9020d-69c8c556d7-8j697"] Sep 16 07:59:56.666300 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.666277 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2"] Sep 16 07:59:56.669826 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:56.669806 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c9020d-67bf7d7775-vblk2"] Sep 16 07:59:57.568503 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:57.568471 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4557169e-72cc-4a7d-b42f-e87a5c1608a3" path="/var/lib/kubelet/pods/4557169e-72cc-4a7d-b42f-e87a5c1608a3/volumes" Sep 16 07:59:57.568883 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:57.568856 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79178c36-2949-484f-b547-aed1a748fd50" path="/var/lib/kubelet/pods/79178c36-2949-484f-b547-aed1a748fd50/volumes" Sep 16 07:59:58.582684 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.582648 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x"] Sep 16 07:59:58.583109 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.582886 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" podUID="e4edb58f-4711-4031-aaee-796423e3567c" containerName="maas-api" containerID="cri-o://2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a" gracePeriod=30 Sep 16 07:59:58.847489 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.847429 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:59:58.902094 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.902063 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kr25\" (UniqueName: \"kubernetes.io/projected/e4edb58f-4711-4031-aaee-796423e3567c-kube-api-access-9kr25\") pod \"e4edb58f-4711-4031-aaee-796423e3567c\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " Sep 16 07:59:58.902268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.902120 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-maas-api-tls\") pod \"e4edb58f-4711-4031-aaee-796423e3567c\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " Sep 16 07:59:58.902268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.902179 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-metrics-certs\") pod \"e4edb58f-4711-4031-aaee-796423e3567c\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " Sep 16 07:59:58.902268 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.902246 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4edb58f-4711-4031-aaee-796423e3567c-openshift-service-ca\") pod \"e4edb58f-4711-4031-aaee-796423e3567c\" (UID: \"e4edb58f-4711-4031-aaee-796423e3567c\") " Sep 16 07:59:58.902678 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.902649 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4edb58f-4711-4031-aaee-796423e3567c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4edb58f-4711-4031-aaee-796423e3567c" (UID: "e4edb58f-4711-4031-aaee-796423e3567c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:59:58.904653 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.904598 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e4edb58f-4711-4031-aaee-796423e3567c" (UID: "e4edb58f-4711-4031-aaee-796423e3567c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:59:58.904768 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.904734 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4edb58f-4711-4031-aaee-796423e3567c" (UID: "e4edb58f-4711-4031-aaee-796423e3567c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:59:58.904908 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:58.904887 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4edb58f-4711-4031-aaee-796423e3567c-kube-api-access-9kr25" (OuterVolumeSpecName: "kube-api-access-9kr25") pod "e4edb58f-4711-4031-aaee-796423e3567c" (UID: "e4edb58f-4711-4031-aaee-796423e3567c"). InnerVolumeSpecName "kube-api-access-9kr25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:59:59.003302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.003268 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:59:59.003302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.003298 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4edb58f-4711-4031-aaee-796423e3567c-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:59:59.003302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.003308 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4edb58f-4711-4031-aaee-796423e3567c-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:59:59.003523 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.003318 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kr25\" (UniqueName: \"kubernetes.io/projected/e4edb58f-4711-4031-aaee-796423e3567c-kube-api-access-9kr25\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 07:59:59.651393 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.651355 2580 generic.go:356] "Generic (PLEG): container finished" podID="e4edb58f-4711-4031-aaee-796423e3567c" containerID="2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a" exitCode=0 Sep 16 07:59:59.651877 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.651405 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" Sep 16 07:59:59.651877 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.651442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" event={"ID":"e4edb58f-4711-4031-aaee-796423e3567c","Type":"ContainerDied","Data":"2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a"} Sep 16 07:59:59.651877 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.651510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x" event={"ID":"e4edb58f-4711-4031-aaee-796423e3567c","Type":"ContainerDied","Data":"66d8ea2b4a43f4d4e76cd6416e9ea1ec2b8586ff2c74007a6f8030a82010dfe4"} Sep 16 07:59:59.651877 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.651546 2580 scope.go:117] "RemoveContainer" containerID="2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a" Sep 16 07:59:59.664273 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.664251 2580 scope.go:117] "RemoveContainer" containerID="2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a" Sep 16 07:59:59.664540 ip-10-0-131-134 kubenswrapper[2580]: E0916 07:59:59.664519 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a\": container with ID starting with 2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a not found: ID does not exist" containerID="2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a" Sep 16 07:59:59.664601 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.664548 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a"} err="failed to get container status \"2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a\": rpc error: code = NotFound desc = could not find container \"2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a\": container with ID starting with 2edd7b8375a2eac9b288bc1511a65b743891bc4ca1853ba3a52b15d885e9693a not found: ID does not exist" Sep 16 07:59:59.671299 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.671275 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x"] Sep 16 07:59:59.677722 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.677697 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-987ea6-7fdf8b954c-9ls6x"] Sep 16 07:59:59.768688 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.768652 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" podUID="588048f5-a6c1-474b-bbac-c15065b7c5bb" containerName="main" containerID="cri-o://6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55" gracePeriod=45 Sep 16 07:59:59.781302 ip-10-0-131-134 kubenswrapper[2580]: I0916 07:59:59.781269 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" podUID="969d293b-61ed-4e14-b7fa-0fc9a2987afb" containerName="main" containerID="cri-o://165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0" gracePeriod=45 Sep 16 08:00:00.058988 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.058963 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 08:00:00.062656 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.062636 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 08:00:00.112835 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.112763 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rpt9\" (UniqueName: \"kubernetes.io/projected/588048f5-a6c1-474b-bbac-c15065b7c5bb-kube-api-access-6rpt9\") pod \"588048f5-a6c1-474b-bbac-c15065b7c5bb\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " Sep 16 08:00:00.112981 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.112856 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-model-cache\") pod \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " Sep 16 08:00:00.112981 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.112893 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-model-cache\") pod \"588048f5-a6c1-474b-bbac-c15065b7c5bb\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " Sep 16 08:00:00.112981 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.112930 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/588048f5-a6c1-474b-bbac-c15065b7c5bb-tls-certs\") pod \"588048f5-a6c1-474b-bbac-c15065b7c5bb\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " Sep 16 08:00:00.112981 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.112966 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tls-certs\") pod \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " Sep 16 08:00:00.113250 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.112998 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-home\") pod \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " Sep 16 08:00:00.113250 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.113035 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-dshm\") pod \"588048f5-a6c1-474b-bbac-c15065b7c5bb\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " Sep 16 08:00:00.113250 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.113080 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tmp-dir\") pod \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " Sep 16 08:00:00.113250 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.113105 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-home\") pod \"588048f5-a6c1-474b-bbac-c15065b7c5bb\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " Sep 16 08:00:00.113250 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.113145 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-dshm\") pod \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " Sep 16 08:00:00.113250 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.113171 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "588048f5-a6c1-474b-bbac-c15065b7c5bb" (UID: "588048f5-a6c1-474b-bbac-c15065b7c5bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.113250 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.113185 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-tmp-dir\") pod \"588048f5-a6c1-474b-bbac-c15065b7c5bb\" (UID: \"588048f5-a6c1-474b-bbac-c15065b7c5bb\") " Sep 16 08:00:00.113698 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.113285 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95qhw\" (UniqueName: \"kubernetes.io/projected/969d293b-61ed-4e14-b7fa-0fc9a2987afb-kube-api-access-95qhw\") pod \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\" (UID: \"969d293b-61ed-4e14-b7fa-0fc9a2987afb\") " Sep 16 08:00:00.113698 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.113643 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-model-cache\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.114462 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.113807 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-model-cache" (OuterVolumeSpecName: "model-cache") pod "969d293b-61ed-4e14-b7fa-0fc9a2987afb" (UID: "969d293b-61ed-4e14-b7fa-0fc9a2987afb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.114462 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.114007 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-home" (OuterVolumeSpecName: "home") pod "588048f5-a6c1-474b-bbac-c15065b7c5bb" (UID: "588048f5-a6c1-474b-bbac-c15065b7c5bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.114462 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.114240 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "969d293b-61ed-4e14-b7fa-0fc9a2987afb" (UID: "969d293b-61ed-4e14-b7fa-0fc9a2987afb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.114462 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.114397 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-home" (OuterVolumeSpecName: "home") pod "969d293b-61ed-4e14-b7fa-0fc9a2987afb" (UID: "969d293b-61ed-4e14-b7fa-0fc9a2987afb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.114462 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.114419 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "588048f5-a6c1-474b-bbac-c15065b7c5bb" (UID: "588048f5-a6c1-474b-bbac-c15065b7c5bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.115523 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.115492 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "969d293b-61ed-4e14-b7fa-0fc9a2987afb" (UID: "969d293b-61ed-4e14-b7fa-0fc9a2987afb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:00.116019 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.115983 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588048f5-a6c1-474b-bbac-c15065b7c5bb-kube-api-access-6rpt9" (OuterVolumeSpecName: "kube-api-access-6rpt9") pod "588048f5-a6c1-474b-bbac-c15065b7c5bb" (UID: "588048f5-a6c1-474b-bbac-c15065b7c5bb"). InnerVolumeSpecName "kube-api-access-6rpt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:00.116290 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.116080 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969d293b-61ed-4e14-b7fa-0fc9a2987afb-kube-api-access-95qhw" (OuterVolumeSpecName: "kube-api-access-95qhw") pod "969d293b-61ed-4e14-b7fa-0fc9a2987afb" (UID: "969d293b-61ed-4e14-b7fa-0fc9a2987afb"). InnerVolumeSpecName "kube-api-access-95qhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:00.117794 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.117767 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-dshm" (OuterVolumeSpecName: "dshm") pod "588048f5-a6c1-474b-bbac-c15065b7c5bb" (UID: "588048f5-a6c1-474b-bbac-c15065b7c5bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.117945 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.117895 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-dshm" (OuterVolumeSpecName: "dshm") pod "969d293b-61ed-4e14-b7fa-0fc9a2987afb" (UID: "969d293b-61ed-4e14-b7fa-0fc9a2987afb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.118293 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.118271 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/588048f5-a6c1-474b-bbac-c15065b7c5bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "588048f5-a6c1-474b-bbac-c15065b7c5bb" (UID: "588048f5-a6c1-474b-bbac-c15065b7c5bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:00.214811 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214776 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95qhw\" (UniqueName: \"kubernetes.io/projected/969d293b-61ed-4e14-b7fa-0fc9a2987afb-kube-api-access-95qhw\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.214811 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214808 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rpt9\" (UniqueName: \"kubernetes.io/projected/588048f5-a6c1-474b-bbac-c15065b7c5bb-kube-api-access-6rpt9\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.215033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214821 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-model-cache\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.215033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214835 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/588048f5-a6c1-474b-bbac-c15065b7c5bb-tls-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.215033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214848 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tls-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.215033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214864 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-home\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.215033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214876 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-dshm\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.215033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214886 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-tmp-dir\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.215033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214896 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-home\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.215033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214906 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969d293b-61ed-4e14-b7fa-0fc9a2987afb-dshm\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.215033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.214916 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588048f5-a6c1-474b-bbac-c15065b7c5bb-tmp-dir\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.658950 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.658918 2580 generic.go:356] "Generic (PLEG): container finished" podID="588048f5-a6c1-474b-bbac-c15065b7c5bb" containerID="6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55" exitCode=0 Sep 16 08:00:00.659382 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.658965 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" Sep 16 08:00:00.659382 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.659003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" event={"ID":"588048f5-a6c1-474b-bbac-c15065b7c5bb","Type":"ContainerDied","Data":"6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55"} Sep 16 08:00:00.659382 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.659047 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s" event={"ID":"588048f5-a6c1-474b-bbac-c15065b7c5bb","Type":"ContainerDied","Data":"29619b016b6de2856d65cd1aec861f3a71a11dbc6b520e87fff7ac6183ae4b01"} Sep 16 08:00:00.659382 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.659071 2580 scope.go:117] "RemoveContainer" containerID="6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55" Sep 16 08:00:00.661873 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.661846 2580 generic.go:356] "Generic (PLEG): container finished" podID="969d293b-61ed-4e14-b7fa-0fc9a2987afb" containerID="165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0" exitCode=0 Sep 16 08:00:00.661873 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.661876 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" Sep 16 08:00:00.662074 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.661928 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" event={"ID":"969d293b-61ed-4e14-b7fa-0fc9a2987afb","Type":"ContainerDied","Data":"165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0"} Sep 16 08:00:00.662074 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.661969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5" event={"ID":"969d293b-61ed-4e14-b7fa-0fc9a2987afb","Type":"ContainerDied","Data":"a2e9ffa45a02fcb7a0b6e8e55ecb1fc07c06c8db20309bbcb7b2ab58e44a13c2"} Sep 16 08:00:00.670013 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.669993 2580 scope.go:117] "RemoveContainer" containerID="6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55" Sep 16 08:00:00.670342 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:00:00.670313 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55\": container with ID starting with 6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55 not found: ID does not exist" containerID="6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55" Sep 16 08:00:00.670443 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.670346 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55"} err="failed to get container status \"6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55\": rpc error: code = NotFound desc = could not find container \"6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55\": container with ID starting with 6672db33b87b784d903299e789c539e202232cc5feebbea58c9409fe158d8b55 not found: ID does not exist" Sep 16 08:00:00.670443 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.670361 2580 scope.go:117] "RemoveContainer" containerID="165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0" Sep 16 08:00:00.679794 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.679771 2580 scope.go:117] "RemoveContainer" containerID="165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0" Sep 16 08:00:00.680112 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:00:00.680086 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0\": container with ID starting with 165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0 not found: ID does not exist" containerID="165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0" Sep 16 08:00:00.680249 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.680121 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0"} err="failed to get container status \"165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0\": rpc error: code = NotFound desc = could not find container \"165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0\": container with ID starting with 165f427167d97602024eaa547393f5fc541954308a63220156f3bcdc53ef65e0 not found: ID does not exist" Sep 16 08:00:00.682822 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.682801 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s"] Sep 16 08:00:00.686581 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.686561 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-w0-c9020d-kserve-7b984dc56f-7f68s"] Sep 16 08:00:00.696772 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.696753 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5"] Sep 16 08:00:00.700804 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:00.700779 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/embedding-w0-c9020d-kserve-75769ddc46-zf9t5"] Sep 16 08:00:01.561741 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:01.561706 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="588048f5-a6c1-474b-bbac-c15065b7c5bb" path="/var/lib/kubelet/pods/588048f5-a6c1-474b-bbac-c15065b7c5bb/volumes" Sep 16 08:00:01.562157 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:01.562142 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="969d293b-61ed-4e14-b7fa-0fc9a2987afb" path="/var/lib/kubelet/pods/969d293b-61ed-4e14-b7fa-0fc9a2987afb/volumes" Sep 16 08:00:01.562529 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:01.562516 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4edb58f-4711-4031-aaee-796423e3567c" path="/var/lib/kubelet/pods/e4edb58f-4711-4031-aaee-796423e3567c/volumes" Sep 16 08:00:15.300287 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:15.300250 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:00:21.880977 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:21.880939 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps"] Sep 16 08:00:21.881580 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:21.881251 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" podUID="63d2c21c-4653-4a06-9c8f-c3b816f76747" containerName="maas-api" containerID="cri-o://9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780" gracePeriod=30 Sep 16 08:00:22.143157 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.143087 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 08:00:22.232084 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.232051 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-maas-api-tls\") pod \"63d2c21c-4653-4a06-9c8f-c3b816f76747\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " Sep 16 08:00:22.232296 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.232128 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htzjh\" (UniqueName: \"kubernetes.io/projected/63d2c21c-4653-4a06-9c8f-c3b816f76747-kube-api-access-htzjh\") pod \"63d2c21c-4653-4a06-9c8f-c3b816f76747\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " Sep 16 08:00:22.232296 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.232154 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d2c21c-4653-4a06-9c8f-c3b816f76747-openshift-service-ca\") pod \"63d2c21c-4653-4a06-9c8f-c3b816f76747\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " Sep 16 08:00:22.232296 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.232248 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-metrics-certs\") pod \"63d2c21c-4653-4a06-9c8f-c3b816f76747\" (UID: \"63d2c21c-4653-4a06-9c8f-c3b816f76747\") " Sep 16 08:00:22.232615 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.232587 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d2c21c-4653-4a06-9c8f-c3b816f76747-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63d2c21c-4653-4a06-9c8f-c3b816f76747" (UID: "63d2c21c-4653-4a06-9c8f-c3b816f76747"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:22.234258 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.234224 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "63d2c21c-4653-4a06-9c8f-c3b816f76747" (UID: "63d2c21c-4653-4a06-9c8f-c3b816f76747"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:22.234376 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.234249 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63d2c21c-4653-4a06-9c8f-c3b816f76747" (UID: "63d2c21c-4653-4a06-9c8f-c3b816f76747"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:22.234376 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.234326 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d2c21c-4653-4a06-9c8f-c3b816f76747-kube-api-access-htzjh" (OuterVolumeSpecName: "kube-api-access-htzjh") pod "63d2c21c-4653-4a06-9c8f-c3b816f76747" (UID: "63d2c21c-4653-4a06-9c8f-c3b816f76747"). InnerVolumeSpecName "kube-api-access-htzjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:22.332956 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.332916 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htzjh\" (UniqueName: \"kubernetes.io/projected/63d2c21c-4653-4a06-9c8f-c3b816f76747-kube-api-access-htzjh\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:22.332956 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.332951 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d2c21c-4653-4a06-9c8f-c3b816f76747-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:22.332956 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.332962 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:22.333194 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.332970 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d2c21c-4653-4a06-9c8f-c3b816f76747-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:22.810060 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.810025 2580 generic.go:356] "Generic (PLEG): container finished" podID="63d2c21c-4653-4a06-9c8f-c3b816f76747" containerID="9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780" exitCode=0 Sep 16 08:00:22.810297 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.810086 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" Sep 16 08:00:22.810297 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.810118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" event={"ID":"63d2c21c-4653-4a06-9c8f-c3b816f76747","Type":"ContainerDied","Data":"9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780"} Sep 16 08:00:22.810297 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.810172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps" event={"ID":"63d2c21c-4653-4a06-9c8f-c3b816f76747","Type":"ContainerDied","Data":"11298b5d977ac77734a079d87ca06de2e13ee101ac57ce5aae09c2c61a7dacfc"} Sep 16 08:00:22.810297 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.810195 2580 scope.go:117] "RemoveContainer" containerID="9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780" Sep 16 08:00:22.820446 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.820426 2580 scope.go:117] "RemoveContainer" containerID="9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780" Sep 16 08:00:22.820779 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:00:22.820758 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780\": container with ID starting with 9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780 not found: ID does not exist" containerID="9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780" Sep 16 08:00:22.820838 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.820788 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780"} err="failed to get container status \"9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780\": rpc error: code = NotFound desc = could not find container \"9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780\": container with ID starting with 9522f25048919367c9326e871c7629aa7235627f80ef7651098d0b9c91c22780 not found: ID does not exist" Sep 16 08:00:22.833247 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.833203 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps"] Sep 16 08:00:22.837287 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:22.837253 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3a5b4c-6786b77dfd-wx2ps"] Sep 16 08:00:23.561821 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:23.561787 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63d2c21c-4653-4a06-9c8f-c3b816f76747" path="/var/lib/kubelet/pods/63d2c21c-4653-4a06-9c8f-c3b816f76747/volumes" Sep 16 08:00:35.225345 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:35.225310 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv"] Sep 16 08:00:39.658865 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.658807 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr"] Sep 16 08:00:39.661127 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.661074 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" podUID="5a0a69e1-a503-4145-b4c6-04784cbfff5c" containerName="maas-api" containerID="cri-o://b75a7e95e660bbb5c2aeb71fc7cf576f0dcfca80ae19e493c75e4360726e2650" gracePeriod=30 Sep 16 08:00:39.717335 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.717300 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:00:39.917990 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.917960 2580 generic.go:356] "Generic (PLEG): container finished" podID="5a0a69e1-a503-4145-b4c6-04784cbfff5c" containerID="b75a7e95e660bbb5c2aeb71fc7cf576f0dcfca80ae19e493c75e4360726e2650" exitCode=0 Sep 16 08:00:39.918105 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.918016 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" event={"ID":"5a0a69e1-a503-4145-b4c6-04784cbfff5c","Type":"ContainerDied","Data":"b75a7e95e660bbb5c2aeb71fc7cf576f0dcfca80ae19e493c75e4360726e2650"} Sep 16 08:00:39.918105 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.918093 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" event={"ID":"5a0a69e1-a503-4145-b4c6-04784cbfff5c","Type":"ContainerDied","Data":"d214203c8edf96bf15a14c3201a821aa0d1c45cc324d32fdc73e08d7052facf2"} Sep 16 08:00:39.918253 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.918121 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d214203c8edf96bf15a14c3201a821aa0d1c45cc324d32fdc73e08d7052facf2" Sep 16 08:00:39.933680 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.933660 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 08:00:39.987662 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.987594 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a69e1-a503-4145-b4c6-04784cbfff5c-openshift-service-ca\") pod \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " Sep 16 08:00:39.987662 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.987640 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-maas-api-tls\") pod \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " Sep 16 08:00:39.987863 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.987670 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djhnr\" (UniqueName: \"kubernetes.io/projected/5a0a69e1-a503-4145-b4c6-04784cbfff5c-kube-api-access-djhnr\") pod \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " Sep 16 08:00:39.987863 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.987749 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-metrics-certs\") pod \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\" (UID: \"5a0a69e1-a503-4145-b4c6-04784cbfff5c\") " Sep 16 08:00:39.988016 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.987989 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0a69e1-a503-4145-b4c6-04784cbfff5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a0a69e1-a503-4145-b4c6-04784cbfff5c" (UID: "5a0a69e1-a503-4145-b4c6-04784cbfff5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:39.988104 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.988089 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a69e1-a503-4145-b4c6-04784cbfff5c-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:39.989899 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.989869 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0a69e1-a503-4145-b4c6-04784cbfff5c-kube-api-access-djhnr" (OuterVolumeSpecName: "kube-api-access-djhnr") pod "5a0a69e1-a503-4145-b4c6-04784cbfff5c" (UID: "5a0a69e1-a503-4145-b4c6-04784cbfff5c"). InnerVolumeSpecName "kube-api-access-djhnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:39.989994 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.989900 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a0a69e1-a503-4145-b4c6-04784cbfff5c" (UID: "5a0a69e1-a503-4145-b4c6-04784cbfff5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:39.989994 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:39.989922 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5a0a69e1-a503-4145-b4c6-04784cbfff5c" (UID: "5a0a69e1-a503-4145-b4c6-04784cbfff5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:40.088625 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:40.088594 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:40.088625 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:40.088621 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djhnr\" (UniqueName: \"kubernetes.io/projected/5a0a69e1-a503-4145-b4c6-04784cbfff5c-kube-api-access-djhnr\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:40.088806 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:40.088636 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0a69e1-a503-4145-b4c6-04784cbfff5c-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:40.924407 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:40.924373 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr" Sep 16 08:00:40.949331 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:40.949303 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr"] Sep 16 08:00:40.954554 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:40.954527 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a243c-684b85d645-2j7gr"] Sep 16 08:00:41.560601 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:41.560564 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a0a69e1-a503-4145-b4c6-04784cbfff5c" path="/var/lib/kubelet/pods/5a0a69e1-a503-4145-b4c6-04784cbfff5c/volumes" Sep 16 08:00:47.313680 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.313643 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks"] Sep 16 08:00:47.314189 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.313909 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" podUID="6d80d240-6d72-4abb-ab72-69992b317fa2" containerName="maas-api" containerID="cri-o://caf0988c2f0f785edfbf5a66b94f034f722a380efc44a552f18c583d146fe110" gracePeriod=30 Sep 16 08:00:47.571842 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.571784 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 08:00:47.644571 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.644543 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-metrics-certs\") pod \"6d80d240-6d72-4abb-ab72-69992b317fa2\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " Sep 16 08:00:47.644716 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.644587 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-maas-api-tls\") pod \"6d80d240-6d72-4abb-ab72-69992b317fa2\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " Sep 16 08:00:47.644716 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.644632 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d80d240-6d72-4abb-ab72-69992b317fa2-openshift-service-ca\") pod \"6d80d240-6d72-4abb-ab72-69992b317fa2\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " Sep 16 08:00:47.644716 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.644658 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bzgz\" (UniqueName: \"kubernetes.io/projected/6d80d240-6d72-4abb-ab72-69992b317fa2-kube-api-access-5bzgz\") pod \"6d80d240-6d72-4abb-ab72-69992b317fa2\" (UID: \"6d80d240-6d72-4abb-ab72-69992b317fa2\") " Sep 16 08:00:47.645075 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.645040 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d80d240-6d72-4abb-ab72-69992b317fa2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d80d240-6d72-4abb-ab72-69992b317fa2" (UID: "6d80d240-6d72-4abb-ab72-69992b317fa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:47.646822 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.646789 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d80d240-6d72-4abb-ab72-69992b317fa2" (UID: "6d80d240-6d72-4abb-ab72-69992b317fa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:47.647143 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.647123 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d80d240-6d72-4abb-ab72-69992b317fa2" (UID: "6d80d240-6d72-4abb-ab72-69992b317fa2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:47.647358 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.647152 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d80d240-6d72-4abb-ab72-69992b317fa2-kube-api-access-5bzgz" (OuterVolumeSpecName: "kube-api-access-5bzgz") pod "6d80d240-6d72-4abb-ab72-69992b317fa2" (UID: "6d80d240-6d72-4abb-ab72-69992b317fa2"). InnerVolumeSpecName "kube-api-access-5bzgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:47.743577 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.743549 2580 scope.go:117] "RemoveContainer" containerID="caf0988c2f0f785edfbf5a66b94f034f722a380efc44a552f18c583d146fe110" Sep 16 08:00:47.746075 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.746050 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d80d240-6d72-4abb-ab72-69992b317fa2-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:47.746075 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.746075 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bzgz\" (UniqueName: \"kubernetes.io/projected/6d80d240-6d72-4abb-ab72-69992b317fa2-kube-api-access-5bzgz\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:47.746237 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.746090 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:47.746237 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.746103 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d80d240-6d72-4abb-ab72-69992b317fa2-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:47.754736 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.754713 2580 scope.go:117] "RemoveContainer" containerID="b75a7e95e660bbb5c2aeb71fc7cf576f0dcfca80ae19e493c75e4360726e2650" Sep 16 08:00:47.967599 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.967553 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" Sep 16 08:00:47.967599 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.967567 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" event={"ID":"6d80d240-6d72-4abb-ab72-69992b317fa2","Type":"ContainerDied","Data":"caf0988c2f0f785edfbf5a66b94f034f722a380efc44a552f18c583d146fe110"} Sep 16 08:00:47.967862 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:47.967663 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks" event={"ID":"6d80d240-6d72-4abb-ab72-69992b317fa2","Type":"ContainerDied","Data":"379f5a04f12a8036d540dd2b26ea475d6236464cfd44a9922aa07e88ac07af6a"} Sep 16 08:00:48.001497 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:48.001467 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks"] Sep 16 08:00:48.005844 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:48.005815 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc27a5-777c77c68c-bz4ks"] Sep 16 08:00:49.562071 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:49.562037 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d80d240-6d72-4abb-ab72-69992b317fa2" path="/var/lib/kubelet/pods/6d80d240-6d72-4abb-ab72-69992b317fa2/volumes" Sep 16 08:00:50.245394 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.245318 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" podUID="56f8dd92-eb73-43c8-aa56-78988fc09b10" containerName="main" containerID="cri-o://919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556" gracePeriod=45 Sep 16 08:00:50.494764 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.494739 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 08:00:50.572375 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.572290 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-home\") pod \"56f8dd92-eb73-43c8-aa56-78988fc09b10\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " Sep 16 08:00:50.572375 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.572345 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-dshm\") pod \"56f8dd92-eb73-43c8-aa56-78988fc09b10\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " Sep 16 08:00:50.572879 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.572385 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k2nt\" (UniqueName: \"kubernetes.io/projected/56f8dd92-eb73-43c8-aa56-78988fc09b10-kube-api-access-2k2nt\") pod \"56f8dd92-eb73-43c8-aa56-78988fc09b10\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " Sep 16 08:00:50.572879 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.572425 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-model-cache\") pod \"56f8dd92-eb73-43c8-aa56-78988fc09b10\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " Sep 16 08:00:50.572879 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.572520 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-tmp-dir\") pod \"56f8dd92-eb73-43c8-aa56-78988fc09b10\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " Sep 16 08:00:50.572879 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.572574 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f8dd92-eb73-43c8-aa56-78988fc09b10-tls-certs\") pod \"56f8dd92-eb73-43c8-aa56-78988fc09b10\" (UID: \"56f8dd92-eb73-43c8-aa56-78988fc09b10\") " Sep 16 08:00:50.572879 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.572609 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-home" (OuterVolumeSpecName: "home") pod "56f8dd92-eb73-43c8-aa56-78988fc09b10" (UID: "56f8dd92-eb73-43c8-aa56-78988fc09b10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.572879 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.572832 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-model-cache" (OuterVolumeSpecName: "model-cache") pod "56f8dd92-eb73-43c8-aa56-78988fc09b10" (UID: "56f8dd92-eb73-43c8-aa56-78988fc09b10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.573205 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.572893 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-home\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.573205 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.572928 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56f8dd92-eb73-43c8-aa56-78988fc09b10" (UID: "56f8dd92-eb73-43c8-aa56-78988fc09b10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.575100 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.575077 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-dshm" (OuterVolumeSpecName: "dshm") pod "56f8dd92-eb73-43c8-aa56-78988fc09b10" (UID: "56f8dd92-eb73-43c8-aa56-78988fc09b10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.575233 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.575158 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f8dd92-eb73-43c8-aa56-78988fc09b10-kube-api-access-2k2nt" (OuterVolumeSpecName: "kube-api-access-2k2nt") pod "56f8dd92-eb73-43c8-aa56-78988fc09b10" (UID: "56f8dd92-eb73-43c8-aa56-78988fc09b10"). InnerVolumeSpecName "kube-api-access-2k2nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:50.575233 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.575157 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f8dd92-eb73-43c8-aa56-78988fc09b10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56f8dd92-eb73-43c8-aa56-78988fc09b10" (UID: "56f8dd92-eb73-43c8-aa56-78988fc09b10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:50.673668 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.673635 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-tmp-dir\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.673668 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.673661 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f8dd92-eb73-43c8-aa56-78988fc09b10-tls-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.673668 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.673670 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-dshm\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.673880 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.673679 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2nt\" (UniqueName: \"kubernetes.io/projected/56f8dd92-eb73-43c8-aa56-78988fc09b10-kube-api-access-2k2nt\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.673880 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.673688 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f8dd92-eb73-43c8-aa56-78988fc09b10-model-cache\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.987521 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.987487 2580 generic.go:356] "Generic (PLEG): container finished" podID="56f8dd92-eb73-43c8-aa56-78988fc09b10" containerID="919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556" exitCode=0 Sep 16 08:00:50.987699 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.987534 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" Sep 16 08:00:50.987699 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.987571 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" event={"ID":"56f8dd92-eb73-43c8-aa56-78988fc09b10","Type":"ContainerDied","Data":"919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556"} Sep 16 08:00:50.987699 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.987618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv" event={"ID":"56f8dd92-eb73-43c8-aa56-78988fc09b10","Type":"ContainerDied","Data":"3b02819c4af59e36224a6325c607f3f32ceae2382db22eb2b30fb999dae9071d"} Sep 16 08:00:50.987699 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.987635 2580 scope.go:117] "RemoveContainer" containerID="919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556" Sep 16 08:00:50.998095 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.998075 2580 scope.go:117] "RemoveContainer" containerID="919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556" Sep 16 08:00:50.998377 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:00:50.998357 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556\": container with ID starting with 919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556 not found: ID does not exist" containerID="919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556" Sep 16 08:00:50.998438 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:50.998384 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556"} err="failed to get container status \"919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556\": rpc error: code = NotFound desc = could not find container \"919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556\": container with ID starting with 919476eeba6756c06c9418816697d438304c4ecc547502e14fc348ff336fe556 not found: ID does not exist" Sep 16 08:00:51.011271 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:51.011247 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv"] Sep 16 08:00:51.016440 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:51.016414 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-w4-fc27a5-kserve-84c67f9985-t4lfv"] Sep 16 08:00:51.560572 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:51.560541 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56f8dd92-eb73-43c8-aa56-78988fc09b10" path="/var/lib/kubelet/pods/56f8dd92-eb73-43c8-aa56-78988fc09b10/volumes" Sep 16 08:00:57.277308 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.277278 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh"] Sep 16 08:00:57.277782 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.277570 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" podUID="2ce7899e-5d2e-4e65-b611-b839f87ace2d" containerName="maas-api" containerID="cri-o://0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257" gracePeriod=30 Sep 16 08:00:57.530274 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.530180 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 08:00:57.635191 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.635155 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-metrics-certs\") pod \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " Sep 16 08:00:57.635401 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.635344 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce7899e-5d2e-4e65-b611-b839f87ace2d-openshift-service-ca\") pod \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " Sep 16 08:00:57.635401 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.635364 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-maas-api-tls\") pod \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " Sep 16 08:00:57.635401 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.635396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkpsc\" (UniqueName: \"kubernetes.io/projected/2ce7899e-5d2e-4e65-b611-b839f87ace2d-kube-api-access-xkpsc\") pod \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\" (UID: \"2ce7899e-5d2e-4e65-b611-b839f87ace2d\") " Sep 16 08:00:57.635712 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.635677 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce7899e-5d2e-4e65-b611-b839f87ace2d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ce7899e-5d2e-4e65-b611-b839f87ace2d" (UID: "2ce7899e-5d2e-4e65-b611-b839f87ace2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:57.635846 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.635821 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce7899e-5d2e-4e65-b611-b839f87ace2d-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:57.637519 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.637487 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce7899e-5d2e-4e65-b611-b839f87ace2d-kube-api-access-xkpsc" (OuterVolumeSpecName: "kube-api-access-xkpsc") pod "2ce7899e-5d2e-4e65-b611-b839f87ace2d" (UID: "2ce7899e-5d2e-4e65-b611-b839f87ace2d"). InnerVolumeSpecName "kube-api-access-xkpsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:57.637776 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.637751 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ce7899e-5d2e-4e65-b611-b839f87ace2d" (UID: "2ce7899e-5d2e-4e65-b611-b839f87ace2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:57.637829 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.637770 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2ce7899e-5d2e-4e65-b611-b839f87ace2d" (UID: "2ce7899e-5d2e-4e65-b611-b839f87ace2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:57.736361 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.736331 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:57.736361 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.736355 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkpsc\" (UniqueName: \"kubernetes.io/projected/2ce7899e-5d2e-4e65-b611-b839f87ace2d-kube-api-access-xkpsc\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:57.736361 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:57.736366 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ce7899e-5d2e-4e65-b611-b839f87ace2d-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:00:58.033601 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:58.033570 2580 generic.go:356] "Generic (PLEG): container finished" podID="2ce7899e-5d2e-4e65-b611-b839f87ace2d" containerID="0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257" exitCode=0 Sep 16 08:00:58.033764 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:58.033615 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" Sep 16 08:00:58.033764 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:58.033665 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" event={"ID":"2ce7899e-5d2e-4e65-b611-b839f87ace2d","Type":"ContainerDied","Data":"0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257"} Sep 16 08:00:58.033764 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:58.033719 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh" event={"ID":"2ce7899e-5d2e-4e65-b611-b839f87ace2d","Type":"ContainerDied","Data":"60f84cc7a2bbc9081e47867a2bbbe714759ed1acbe71f6311855d3ed501784f3"} Sep 16 08:00:58.033764 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:58.033742 2580 scope.go:117] "RemoveContainer" containerID="0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257" Sep 16 08:00:58.044148 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:58.044129 2580 scope.go:117] "RemoveContainer" containerID="0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257" Sep 16 08:00:58.044442 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:00:58.044424 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257\": container with ID starting with 0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257 not found: ID does not exist" containerID="0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257" Sep 16 08:00:58.044519 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:58.044454 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257"} err="failed to get container status \"0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257\": rpc error: code = NotFound desc = could not find container \"0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257\": container with ID starting with 0d57e2a347700af1b2cf71a383ac5d69654c73b963d60911efd0b6e829b9b257 not found: ID does not exist" Sep 16 08:00:58.058097 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:58.058075 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh"] Sep 16 08:00:58.060461 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:58.060442 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-984c1a-59f78447f7-4nghh"] Sep 16 08:00:59.560305 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:00:59.560272 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ce7899e-5d2e-4e65-b611-b839f87ace2d" path="/var/lib/kubelet/pods/2ce7899e-5d2e-4e65-b611-b839f87ace2d/volumes" Sep 16 08:01:02.808028 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:02.807999 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:01:18.805111 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:18.805076 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:01:32.602953 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:32.602919 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:01:33.869933 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.869895 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-e449b8"] Sep 16 08:01:33.870304 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870282 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4557169e-72cc-4a7d-b42f-e87a5c1608a3" containerName="maas-api" Sep 16 08:01:33.870304 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870295 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4557169e-72cc-4a7d-b42f-e87a5c1608a3" containerName="maas-api" Sep 16 08:01:33.870304 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870305 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969d293b-61ed-4e14-b7fa-0fc9a2987afb" containerName="main" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870311 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="969d293b-61ed-4e14-b7fa-0fc9a2987afb" containerName="main" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870321 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d80d240-6d72-4abb-ab72-69992b317fa2" containerName="maas-api" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870326 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d80d240-6d72-4abb-ab72-69992b317fa2" containerName="maas-api" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870334 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63d2c21c-4653-4a06-9c8f-c3b816f76747" containerName="maas-api" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870339 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63d2c21c-4653-4a06-9c8f-c3b816f76747" containerName="maas-api" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870352 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79178c36-2949-484f-b547-aed1a748fd50" containerName="payload-processing" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870357 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79178c36-2949-484f-b547-aed1a748fd50" containerName="payload-processing" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870363 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce7899e-5d2e-4e65-b611-b839f87ace2d" containerName="maas-api" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870368 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce7899e-5d2e-4e65-b611-b839f87ace2d" containerName="maas-api" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870374 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4edb58f-4711-4031-aaee-796423e3567c" containerName="maas-api" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870379 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4edb58f-4711-4031-aaee-796423e3567c" containerName="maas-api" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870389 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0a69e1-a503-4145-b4c6-04784cbfff5c" containerName="maas-api" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870395 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0a69e1-a503-4145-b4c6-04784cbfff5c" containerName="maas-api" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870405 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="588048f5-a6c1-474b-bbac-c15065b7c5bb" containerName="main" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870410 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="588048f5-a6c1-474b-bbac-c15065b7c5bb" containerName="main" Sep 16 08:01:33.870412 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870415 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56f8dd92-eb73-43c8-aa56-78988fc09b10" containerName="main" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870420 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56f8dd92-eb73-43c8-aa56-78988fc09b10" containerName="main" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870473 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0a69e1-a503-4145-b4c6-04784cbfff5c" containerName="maas-api" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870482 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="969d293b-61ed-4e14-b7fa-0fc9a2987afb" containerName="main" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870488 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4edb58f-4711-4031-aaee-796423e3567c" containerName="maas-api" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870493 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="588048f5-a6c1-474b-bbac-c15065b7c5bb" containerName="main" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870498 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce7899e-5d2e-4e65-b611-b839f87ace2d" containerName="maas-api" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870504 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="63d2c21c-4653-4a06-9c8f-c3b816f76747" containerName="maas-api" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870511 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4557169e-72cc-4a7d-b42f-e87a5c1608a3" containerName="maas-api" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870516 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79178c36-2949-484f-b547-aed1a748fd50" containerName="payload-processing" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870521 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d80d240-6d72-4abb-ab72-69992b317fa2" containerName="maas-api" Sep 16 08:01:33.870900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.870526 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56f8dd92-eb73-43c8-aa56-78988fc09b10" containerName="main" Sep 16 08:01:33.875085 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.874877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-e449b8" Sep 16 08:01:33.877667 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.877643 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kvkmt\"" Sep 16 08:01:33.878407 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.878383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-e449b8"] Sep 16 08:01:33.938164 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:33.938137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdf5\" (UniqueName: \"kubernetes.io/projected/3b2c6c10-ddbd-4e83-b160-4bf5015af3df-kube-api-access-6sdf5\") pod \"test-curl-7696-e449b8\" (UID: \"3b2c6c10-ddbd-4e83-b160-4bf5015af3df\") " pod="opendatahub/test-curl-7696-e449b8" Sep 16 08:01:34.039178 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:34.039146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdf5\" (UniqueName: \"kubernetes.io/projected/3b2c6c10-ddbd-4e83-b160-4bf5015af3df-kube-api-access-6sdf5\") pod \"test-curl-7696-e449b8\" (UID: \"3b2c6c10-ddbd-4e83-b160-4bf5015af3df\") " pod="opendatahub/test-curl-7696-e449b8" Sep 16 08:01:34.048018 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:34.047996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdf5\" (UniqueName: \"kubernetes.io/projected/3b2c6c10-ddbd-4e83-b160-4bf5015af3df-kube-api-access-6sdf5\") pod \"test-curl-7696-e449b8\" (UID: \"3b2c6c10-ddbd-4e83-b160-4bf5015af3df\") " pod="opendatahub/test-curl-7696-e449b8" Sep 16 08:01:34.185931 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:34.185905 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-e449b8" Sep 16 08:01:34.454264 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:34.454239 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-e449b8"] Sep 16 08:01:34.455653 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:01:34.455627 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b2c6c10_ddbd_4e83_b160_4bf5015af3df.slice/crio-5eb6c7d47293d49e1362acc26f468ab96315b55b0a9f5e11f87f3f227ac3b2bc WatchSource:0}: Error finding container 5eb6c7d47293d49e1362acc26f468ab96315b55b0a9f5e11f87f3f227ac3b2bc: Status 404 returned error can't find the container with id 5eb6c7d47293d49e1362acc26f468ab96315b55b0a9f5e11f87f3f227ac3b2bc Sep 16 08:01:35.250470 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:35.250434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-e449b8" event={"ID":"3b2c6c10-ddbd-4e83-b160-4bf5015af3df","Type":"ContainerStarted","Data":"f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef"} Sep 16 08:01:35.250822 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:35.250479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-e449b8" event={"ID":"3b2c6c10-ddbd-4e83-b160-4bf5015af3df","Type":"ContainerStarted","Data":"5eb6c7d47293d49e1362acc26f468ab96315b55b0a9f5e11f87f3f227ac3b2bc"} Sep 16 08:01:35.266652 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:35.266596 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7696-e449b8" podStartSLOduration=2.266577991 podStartE2EDuration="2.266577991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:01:35.264884608 +0000 UTC m=+1668.184500555" watchObservedRunningTime="2026-09-16 08:01:35.266577991 +0000 UTC m=+1668.186194145" Sep 16 08:01:35.860319 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:35.860287 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-e449b8"] Sep 16 08:01:35.864665 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:35.864627 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-e449b8"] Sep 16 08:01:36.256229 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:01:36.256185 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7696-e449b8" container="test-curl-7696-e449b8" Sep 16 08:01:37.264756 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:37.264714 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7696-e449b8" podUID="3b2c6c10-ddbd-4e83-b160-4bf5015af3df" containerName="test-curl-7696-e449b8" containerID="cri-o://f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef" gracePeriod=2 Sep 16 08:01:37.267174 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:01:37.267146 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-e449b8\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="3b2c6c10-ddbd-4e83-b160-4bf5015af3df" pod="opendatahub/test-curl-7696-e449b8" Sep 16 08:01:37.560575 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:01:37.560486 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-e449b8\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="3b2c6c10-ddbd-4e83-b160-4bf5015af3df" pod="opendatahub/test-curl-7696-e449b8" Sep 16 08:01:39.439750 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:39.439723 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-e449b8" Sep 16 08:01:39.486767 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:39.486742 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sdf5\" (UniqueName: \"kubernetes.io/projected/3b2c6c10-ddbd-4e83-b160-4bf5015af3df-kube-api-access-6sdf5\") pod \"3b2c6c10-ddbd-4e83-b160-4bf5015af3df\" (UID: \"3b2c6c10-ddbd-4e83-b160-4bf5015af3df\") " Sep 16 08:01:39.488679 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:39.488660 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2c6c10-ddbd-4e83-b160-4bf5015af3df-kube-api-access-6sdf5" (OuterVolumeSpecName: "kube-api-access-6sdf5") pod "3b2c6c10-ddbd-4e83-b160-4bf5015af3df" (UID: "3b2c6c10-ddbd-4e83-b160-4bf5015af3df"). InnerVolumeSpecName "kube-api-access-6sdf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:01:39.560885 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:39.560830 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b2c6c10-ddbd-4e83-b160-4bf5015af3df" path="/var/lib/kubelet/pods/3b2c6c10-ddbd-4e83-b160-4bf5015af3df/volumes" Sep 16 08:01:39.588244 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:39.588195 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdf5\" (UniqueName: \"kubernetes.io/projected/3b2c6c10-ddbd-4e83-b160-4bf5015af3df-kube-api-access-6sdf5\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:01:40.284372 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:40.284333 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b2c6c10-ddbd-4e83-b160-4bf5015af3df" containerID="f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef" exitCode=137 Sep 16 08:01:40.284372 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:40.284374 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-e449b8" Sep 16 08:01:40.284587 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:40.284437 2580 scope.go:117] "RemoveContainer" containerID="f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef" Sep 16 08:01:40.293993 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:40.293974 2580 scope.go:117] "RemoveContainer" containerID="f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef" Sep 16 08:01:40.294303 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:01:40.294281 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef\": container with ID starting with f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef not found: ID does not exist" containerID="f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef" Sep 16 08:01:40.294350 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:40.294311 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef"} err="failed to get container status \"f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef\": rpc error: code = NotFound desc = could not find container \"f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef\": container with ID starting with f95ec988ec94596ce0821bb9e0b420ac4fe4d5f8d1f2fd80443701eb21d739ef not found: ID does not exist" Sep 16 08:01:42.204907 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:42.204870 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:01:51.400032 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:01:51.399997 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:02:27.912306 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:27.912271 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:02:46.611705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:46.611666 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:02:57.102821 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.102788 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-3a2d26"] Sep 16 08:02:57.103232 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.103188 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2c6c10-ddbd-4e83-b160-4bf5015af3df" containerName="test-curl-7696-e449b8" Sep 16 08:02:57.103232 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.103200 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2c6c10-ddbd-4e83-b160-4bf5015af3df" containerName="test-curl-7696-e449b8" Sep 16 08:02:57.103326 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.103284 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2c6c10-ddbd-4e83-b160-4bf5015af3df" containerName="test-curl-7696-e449b8" Sep 16 08:02:57.106073 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.106055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-3a2d26" Sep 16 08:02:57.108851 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.108825 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kvkmt\"" Sep 16 08:02:57.110421 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.110396 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-3a2d26"] Sep 16 08:02:57.219957 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.219922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjnd\" (UniqueName: \"kubernetes.io/projected/340d8565-b0c5-420a-bf81-4ec3ead42d28-kube-api-access-sbjnd\") pod \"test-curl-7696-3a2d26\" (UID: \"340d8565-b0c5-420a-bf81-4ec3ead42d28\") " pod="opendatahub/test-curl-7696-3a2d26" Sep 16 08:02:57.321379 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.321341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjnd\" (UniqueName: \"kubernetes.io/projected/340d8565-b0c5-420a-bf81-4ec3ead42d28-kube-api-access-sbjnd\") pod \"test-curl-7696-3a2d26\" (UID: \"340d8565-b0c5-420a-bf81-4ec3ead42d28\") " pod="opendatahub/test-curl-7696-3a2d26" Sep 16 08:02:57.329936 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.329907 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjnd\" (UniqueName: \"kubernetes.io/projected/340d8565-b0c5-420a-bf81-4ec3ead42d28-kube-api-access-sbjnd\") pod \"test-curl-7696-3a2d26\" (UID: \"340d8565-b0c5-420a-bf81-4ec3ead42d28\") " pod="opendatahub/test-curl-7696-3a2d26" Sep 16 08:02:57.416379 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.416352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-3a2d26" Sep 16 08:02:57.684053 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.684029 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-3a2d26"] Sep 16 08:02:57.686111 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:02:57.686074 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod340d8565_b0c5_420a_bf81_4ec3ead42d28.slice/crio-8d90d3a27ef20325c8195d6de03faeb153451cfb3493991b4bc5170203de5a93 WatchSource:0}: Error finding container 8d90d3a27ef20325c8195d6de03faeb153451cfb3493991b4bc5170203de5a93: Status 404 returned error can't find the container with id 8d90d3a27ef20325c8195d6de03faeb153451cfb3493991b4bc5170203de5a93 Sep 16 08:02:57.717516 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:57.717489 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-3a2d26" event={"ID":"340d8565-b0c5-420a-bf81-4ec3ead42d28","Type":"ContainerStarted","Data":"8d90d3a27ef20325c8195d6de03faeb153451cfb3493991b4bc5170203de5a93"} Sep 16 08:02:58.725143 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:58.725108 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-3a2d26" event={"ID":"340d8565-b0c5-420a-bf81-4ec3ead42d28","Type":"ContainerStarted","Data":"4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742"} Sep 16 08:02:58.740992 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:58.740949 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7696-3a2d26" podStartSLOduration=1.740935288 podStartE2EDuration="1.740935288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:02:58.739806348 +0000 UTC m=+1751.659422294" watchObservedRunningTime="2026-09-16 08:02:58.740935288 +0000 UTC m=+1751.660551235" Sep 16 08:02:59.352819 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.352766 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-3a2d26"] Sep 16 08:02:59.355726 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.355693 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-3a2d26"] Sep 16 08:02:59.471768 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.471733 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-3eb6fe"] Sep 16 08:02:59.472159 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.472146 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340d8565-b0c5-420a-bf81-4ec3ead42d28" containerName="test-curl-7696-3a2d26" Sep 16 08:02:59.472225 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.472161 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="340d8565-b0c5-420a-bf81-4ec3ead42d28" containerName="test-curl-7696-3a2d26" Sep 16 08:02:59.472271 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.472234 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="340d8565-b0c5-420a-bf81-4ec3ead42d28" containerName="test-curl-7696-3a2d26" Sep 16 08:02:59.482397 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.482367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-3eb6fe" Sep 16 08:02:59.483798 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.483771 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-3eb6fe"] Sep 16 08:02:59.539509 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.539478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4x8\" (UniqueName: \"kubernetes.io/projected/e6996987-e245-4a80-bd9e-a5ed47970b7b-kube-api-access-jn4x8\") pod \"test-curl-7696-3eb6fe\" (UID: \"e6996987-e245-4a80-bd9e-a5ed47970b7b\") " pod="opendatahub/test-curl-7696-3eb6fe" Sep 16 08:02:59.641253 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.641198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4x8\" (UniqueName: \"kubernetes.io/projected/e6996987-e245-4a80-bd9e-a5ed47970b7b-kube-api-access-jn4x8\") pod \"test-curl-7696-3eb6fe\" (UID: \"e6996987-e245-4a80-bd9e-a5ed47970b7b\") " pod="opendatahub/test-curl-7696-3eb6fe" Sep 16 08:02:59.649966 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.649942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4x8\" (UniqueName: \"kubernetes.io/projected/e6996987-e245-4a80-bd9e-a5ed47970b7b-kube-api-access-jn4x8\") pod \"test-curl-7696-3eb6fe\" (UID: \"e6996987-e245-4a80-bd9e-a5ed47970b7b\") " pod="opendatahub/test-curl-7696-3eb6fe" Sep 16 08:02:59.730647 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:02:59.730580 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7696-3a2d26" container="test-curl-7696-3a2d26" Sep 16 08:02:59.793017 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:02:59.792993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-3eb6fe" Sep 16 08:03:00.064600 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:00.064575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-3eb6fe"] Sep 16 08:03:00.065829 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:03:00.065799 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6996987_e245_4a80_bd9e_a5ed47970b7b.slice/crio-dbb389e2825cee6178f82ed30fd5d944a9d7550654d9804de6ed5554903298b1 WatchSource:0}: Error finding container dbb389e2825cee6178f82ed30fd5d944a9d7550654d9804de6ed5554903298b1: Status 404 returned error can't find the container with id dbb389e2825cee6178f82ed30fd5d944a9d7550654d9804de6ed5554903298b1 Sep 16 08:03:00.737271 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:00.737235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-3eb6fe" event={"ID":"e6996987-e245-4a80-bd9e-a5ed47970b7b","Type":"ContainerStarted","Data":"8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b"} Sep 16 08:03:00.737677 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:00.737281 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-3eb6fe" event={"ID":"e6996987-e245-4a80-bd9e-a5ed47970b7b","Type":"ContainerStarted","Data":"dbb389e2825cee6178f82ed30fd5d944a9d7550654d9804de6ed5554903298b1"} Sep 16 08:03:00.737677 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:00.737326 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7696-3a2d26" podUID="340d8565-b0c5-420a-bf81-4ec3ead42d28" containerName="test-curl-7696-3a2d26" containerID="cri-o://4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742" gracePeriod=2 Sep 16 08:03:00.752995 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:00.752948 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7696-3eb6fe" podStartSLOduration=1.7529337900000002 podStartE2EDuration="1.75293379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:03:00.752619482 +0000 UTC m=+1753.672235429" watchObservedRunningTime="2026-09-16 08:03:00.75293379 +0000 UTC m=+1753.672549735" Sep 16 08:03:01.364969 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:01.364932 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-3eb6fe"] Sep 16 08:03:01.368187 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:01.368157 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-3eb6fe"] Sep 16 08:03:01.742385 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:03:01.742359 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7696-3eb6fe" container="test-curl-7696-3eb6fe" Sep 16 08:03:02.107192 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:02.107107 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:03:02.748119 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:02.748085 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7696-3eb6fe" podUID="e6996987-e245-4a80-bd9e-a5ed47970b7b" containerName="test-curl-7696-3eb6fe" containerID="cri-o://8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b" gracePeriod=2 Sep 16 08:03:02.877701 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:02.877676 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-3a2d26" Sep 16 08:03:02.972736 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:02.972706 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbjnd\" (UniqueName: \"kubernetes.io/projected/340d8565-b0c5-420a-bf81-4ec3ead42d28-kube-api-access-sbjnd\") pod \"340d8565-b0c5-420a-bf81-4ec3ead42d28\" (UID: \"340d8565-b0c5-420a-bf81-4ec3ead42d28\") " Sep 16 08:03:02.974710 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:02.974687 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340d8565-b0c5-420a-bf81-4ec3ead42d28-kube-api-access-sbjnd" (OuterVolumeSpecName: "kube-api-access-sbjnd") pod "340d8565-b0c5-420a-bf81-4ec3ead42d28" (UID: "340d8565-b0c5-420a-bf81-4ec3ead42d28"). InnerVolumeSpecName "kube-api-access-sbjnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:03:03.073899 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:03.073830 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbjnd\" (UniqueName: \"kubernetes.io/projected/340d8565-b0c5-420a-bf81-4ec3ead42d28-kube-api-access-sbjnd\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:03:03.560554 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:03.560521 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="340d8565-b0c5-420a-bf81-4ec3ead42d28" path="/var/lib/kubelet/pods/340d8565-b0c5-420a-bf81-4ec3ead42d28/volumes" Sep 16 08:03:03.755420 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:03.755383 2580 generic.go:356] "Generic (PLEG): container finished" podID="340d8565-b0c5-420a-bf81-4ec3ead42d28" containerID="4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742" exitCode=137 Sep 16 08:03:03.755828 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:03.755436 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-3a2d26" Sep 16 08:03:03.755828 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:03.755475 2580 scope.go:117] "RemoveContainer" containerID="4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742" Sep 16 08:03:03.764583 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:03.764558 2580 scope.go:117] "RemoveContainer" containerID="4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742" Sep 16 08:03:03.764847 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:03:03.764830 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742\": container with ID starting with 4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742 not found: ID does not exist" containerID="4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742" Sep 16 08:03:03.764898 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:03.764854 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742"} err="failed to get container status \"4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742\": rpc error: code = NotFound desc = could not find container \"4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742\": container with ID starting with 4286bf5eea23cd0dfb8395b19f33dcd5af3eeb88c21b4694cc70d7d5020f2742 not found: ID does not exist" Sep 16 08:03:04.936948 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:04.936924 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-3eb6fe" Sep 16 08:03:04.997051 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:04.997025 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn4x8\" (UniqueName: \"kubernetes.io/projected/e6996987-e245-4a80-bd9e-a5ed47970b7b-kube-api-access-jn4x8\") pod \"e6996987-e245-4a80-bd9e-a5ed47970b7b\" (UID: \"e6996987-e245-4a80-bd9e-a5ed47970b7b\") " Sep 16 08:03:04.998980 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:04.998953 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6996987-e245-4a80-bd9e-a5ed47970b7b-kube-api-access-jn4x8" (OuterVolumeSpecName: "kube-api-access-jn4x8") pod "e6996987-e245-4a80-bd9e-a5ed47970b7b" (UID: "e6996987-e245-4a80-bd9e-a5ed47970b7b"). InnerVolumeSpecName "kube-api-access-jn4x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:03:05.097912 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:05.097880 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn4x8\" (UniqueName: \"kubernetes.io/projected/e6996987-e245-4a80-bd9e-a5ed47970b7b-kube-api-access-jn4x8\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:03:05.560266 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:05.560232 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6996987-e245-4a80-bd9e-a5ed47970b7b" path="/var/lib/kubelet/pods/e6996987-e245-4a80-bd9e-a5ed47970b7b/volumes" Sep 16 08:03:05.770570 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:05.770536 2580 generic.go:356] "Generic (PLEG): container finished" podID="e6996987-e245-4a80-bd9e-a5ed47970b7b" containerID="8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b" exitCode=137 Sep 16 08:03:05.770725 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:05.770580 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-3eb6fe" Sep 16 08:03:05.770725 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:05.770592 2580 scope.go:117] "RemoveContainer" containerID="8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b" Sep 16 08:03:05.778911 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:05.778898 2580 scope.go:117] "RemoveContainer" containerID="8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b" Sep 16 08:03:05.779177 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:03:05.779154 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b\": container with ID starting with 8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b not found: ID does not exist" containerID="8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b" Sep 16 08:03:05.779237 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:05.779186 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b"} err="failed to get container status \"8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b\": rpc error: code = NotFound desc = could not find container \"8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b\": container with ID starting with 8112ff084aa277fc4d41379be4f8c531c259ff237377b46b78d1a002177f0d1b not found: ID does not exist" Sep 16 08:03:13.196847 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:13.196795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:03:16.373595 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.373564 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg"] Sep 16 08:03:16.373988 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.373975 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6996987-e245-4a80-bd9e-a5ed47970b7b" containerName="test-curl-7696-3eb6fe" Sep 16 08:03:16.374033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.373989 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6996987-e245-4a80-bd9e-a5ed47970b7b" containerName="test-curl-7696-3eb6fe" Sep 16 08:03:16.374065 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.374057 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6996987-e245-4a80-bd9e-a5ed47970b7b" containerName="test-curl-7696-3eb6fe" Sep 16 08:03:16.378355 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.378335 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.381313 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.381291 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"openshift-service-ca.crt\"" Sep 16 08:03:16.382421 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.382398 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"kube-root-ca.crt\"" Sep 16 08:03:16.382557 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.382528 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"rate-test-model-3616cc-kserve-self-signed-certs\"" Sep 16 08:03:16.382684 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.382582 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"default-dockercfg-lm9f7\"" Sep 16 08:03:16.393237 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.388801 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg"] Sep 16 08:03:16.494543 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.494513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-tmp-dir\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.494657 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.494550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rms6\" (UniqueName: \"kubernetes.io/projected/d5e84e06-3163-4450-8cc7-278d239e6187-kube-api-access-9rms6\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.494657 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.494572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-home\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.494819 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.494661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-dshm\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.494819 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.494688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-model-cache\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.494819 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.494765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e84e06-3163-4450-8cc7-278d239e6187-tls-certs\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.595404 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.595374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e84e06-3163-4450-8cc7-278d239e6187-tls-certs\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.595524 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.595420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-tmp-dir\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.595524 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.595454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rms6\" (UniqueName: \"kubernetes.io/projected/d5e84e06-3163-4450-8cc7-278d239e6187-kube-api-access-9rms6\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.595524 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.595490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-home\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.595683 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.595543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-dshm\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.595683 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.595569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-model-cache\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.595909 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.595885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-tmp-dir\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.595909 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.595898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-home\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.596020 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.595975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-model-cache\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.597694 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.597671 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-dshm\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.597891 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.597876 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e84e06-3163-4450-8cc7-278d239e6187-tls-certs\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.604827 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.604805 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rms6\" (UniqueName: \"kubernetes.io/projected/d5e84e06-3163-4450-8cc7-278d239e6187-kube-api-access-9rms6\") pod \"rate-test-model-3616cc-kserve-6f7b895b4c-th5gg\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.694367 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.694341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:16.968048 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:16.968017 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg"] Sep 16 08:03:16.968828 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:03:16.968801 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5e84e06_3163_4450_8cc7_278d239e6187.slice/crio-3f494ddd642dd2e84ec7f5aa616f3eca57ee2de0dadf5d4d36e78b7f5d0a0a50 WatchSource:0}: Error finding container 3f494ddd642dd2e84ec7f5aa616f3eca57ee2de0dadf5d4d36e78b7f5d0a0a50: Status 404 returned error can't find the container with id 3f494ddd642dd2e84ec7f5aa616f3eca57ee2de0dadf5d4d36e78b7f5d0a0a50 Sep 16 08:03:17.839547 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:17.839506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" event={"ID":"d5e84e06-3163-4450-8cc7-278d239e6187","Type":"ContainerStarted","Data":"0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4"} Sep 16 08:03:17.840040 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:17.839561 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" event={"ID":"d5e84e06-3163-4450-8cc7-278d239e6187","Type":"ContainerStarted","Data":"3f494ddd642dd2e84ec7f5aa616f3eca57ee2de0dadf5d4d36e78b7f5d0a0a50"} Sep 16 08:03:17.856157 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:17.856111 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" podStartSLOduration=1.856094021 podStartE2EDuration="1.856094021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:03:17.854642514 +0000 UTC m=+1770.774258461" watchObservedRunningTime="2026-09-16 08:03:17.856094021 +0000 UTC m=+1770.775709967" Sep 16 08:03:19.699056 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:19.699017 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:03:26.694669 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:26.694634 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:26.695067 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:26.694682 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:26.707277 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:26.707252 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:26.902447 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:26.902420 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:03:46.403768 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:46.403729 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:03:47.643502 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:47.643474 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 08:03:47.649891 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:03:47.649865 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 08:04:02.207992 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:02.207955 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:04:17.707145 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:17.707105 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:04:22.784419 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:22.784384 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg"] Sep 16 08:04:28.413958 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:28.413913 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:04:29.245435 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.245392 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq"] Sep 16 08:04:29.249821 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.249800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.252382 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.252346 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-796e11-serving-cert\"" Sep 16 08:04:29.257006 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.256982 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq"] Sep 16 08:04:29.323457 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.323410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccea580a-db9b-46a5-965c-fe7421b4ee00-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.323457 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.323467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-metrics-certs\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.323646 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.323511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-maas-api-tls\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.323646 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.323591 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjsj\" (UniqueName: \"kubernetes.io/projected/ccea580a-db9b-46a5-965c-fe7421b4ee00-kube-api-access-7pjsj\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.424871 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.424836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjsj\" (UniqueName: \"kubernetes.io/projected/ccea580a-db9b-46a5-965c-fe7421b4ee00-kube-api-access-7pjsj\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.425261 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.424920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccea580a-db9b-46a5-965c-fe7421b4ee00-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.425261 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.424956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-metrics-certs\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.425261 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.424983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-maas-api-tls\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.425261 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:04:29.425083 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-serving-cert: secret "maas-api-e2e-ipp-a-796e11-serving-cert" not found Sep 16 08:04:29.425261 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:04:29.425154 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-maas-api-tls podName:ccea580a-db9b-46a5-965c-fe7421b4ee00 nodeName:}" failed. No retries permitted until 2026-09-16 08:04:29.925135125 +0000 UTC m=+1842.844751051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-maas-api-tls") pod "maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" (UID: "ccea580a-db9b-46a5-965c-fe7421b4ee00") : secret "maas-api-e2e-ipp-a-796e11-serving-cert" not found Sep 16 08:04:29.425619 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.425600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccea580a-db9b-46a5-965c-fe7421b4ee00-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.427469 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.427441 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-metrics-certs\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.433634 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.433600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjsj\" (UniqueName: \"kubernetes.io/projected/ccea580a-db9b-46a5-965c-fe7421b4ee00-kube-api-access-7pjsj\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.929697 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.929661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-maas-api-tls\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:29.932097 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:29.932065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-maas-api-tls\") pod \"maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:30.162711 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:30.162675 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:30.433342 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:30.433313 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq"] Sep 16 08:04:30.434886 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:04:30.434851 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccea580a_db9b_46a5_965c_fe7421b4ee00.slice/crio-48d7636a16e351cc2ee2a61de96217a2063394408f40a3eaba8f5c0a3bb016a9 WatchSource:0}: Error finding container 48d7636a16e351cc2ee2a61de96217a2063394408f40a3eaba8f5c0a3bb016a9: Status 404 returned error can't find the container with id 48d7636a16e351cc2ee2a61de96217a2063394408f40a3eaba8f5c0a3bb016a9 Sep 16 08:04:30.436637 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:30.436615 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 08:04:31.261432 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:31.261390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" event={"ID":"ccea580a-db9b-46a5-965c-fe7421b4ee00","Type":"ContainerStarted","Data":"bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e"} Sep 16 08:04:31.261635 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:31.261473 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:31.261635 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:31.261500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" event={"ID":"ccea580a-db9b-46a5-965c-fe7421b4ee00","Type":"ContainerStarted","Data":"48d7636a16e351cc2ee2a61de96217a2063394408f40a3eaba8f5c0a3bb016a9"} Sep 16 08:04:31.278053 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:31.278000 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" podStartSLOduration=1.818016705 podStartE2EDuration="2.277984188s" totalImagesPullingTime="459.967483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:04:31.276874176 +0000 UTC m=+1844.196490122" watchObservedRunningTime="2026-09-16 08:04:31.277984188 +0000 UTC m=+1844.197600134" Sep 16 08:04:37.272769 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:37.272738 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:04:37.804002 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:37.803966 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" podUID="d5e84e06-3163-4450-8cc7-278d239e6187" containerName="main" containerID="cri-o://0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4" gracePeriod=45 Sep 16 08:04:38.063002 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.062947 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:04:38.100373 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.100346 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-tmp-dir\") pod \"d5e84e06-3163-4450-8cc7-278d239e6187\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " Sep 16 08:04:38.100510 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.100388 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rms6\" (UniqueName: \"kubernetes.io/projected/d5e84e06-3163-4450-8cc7-278d239e6187-kube-api-access-9rms6\") pod \"d5e84e06-3163-4450-8cc7-278d239e6187\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " Sep 16 08:04:38.100510 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.100424 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-dshm\") pod \"d5e84e06-3163-4450-8cc7-278d239e6187\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " Sep 16 08:04:38.100510 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.100440 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-model-cache\") pod \"d5e84e06-3163-4450-8cc7-278d239e6187\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " Sep 16 08:04:38.100510 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.100464 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e84e06-3163-4450-8cc7-278d239e6187-tls-certs\") pod \"d5e84e06-3163-4450-8cc7-278d239e6187\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " Sep 16 08:04:38.100510 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.100491 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-home\") pod \"d5e84e06-3163-4450-8cc7-278d239e6187\" (UID: \"d5e84e06-3163-4450-8cc7-278d239e6187\") " Sep 16 08:04:38.100788 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.100660 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5e84e06-3163-4450-8cc7-278d239e6187" (UID: "d5e84e06-3163-4450-8cc7-278d239e6187"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:04:38.100788 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.100722 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5e84e06-3163-4450-8cc7-278d239e6187" (UID: "d5e84e06-3163-4450-8cc7-278d239e6187"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:04:38.100788 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.100784 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-tmp-dir\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:04:38.100941 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.100876 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-home" (OuterVolumeSpecName: "home") pod "d5e84e06-3163-4450-8cc7-278d239e6187" (UID: "d5e84e06-3163-4450-8cc7-278d239e6187"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:04:38.102608 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.102581 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-dshm" (OuterVolumeSpecName: "dshm") pod "d5e84e06-3163-4450-8cc7-278d239e6187" (UID: "d5e84e06-3163-4450-8cc7-278d239e6187"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:04:38.102756 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.102726 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e84e06-3163-4450-8cc7-278d239e6187-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5e84e06-3163-4450-8cc7-278d239e6187" (UID: "d5e84e06-3163-4450-8cc7-278d239e6187"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:04:38.102870 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.102845 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e84e06-3163-4450-8cc7-278d239e6187-kube-api-access-9rms6" (OuterVolumeSpecName: "kube-api-access-9rms6") pod "d5e84e06-3163-4450-8cc7-278d239e6187" (UID: "d5e84e06-3163-4450-8cc7-278d239e6187"). InnerVolumeSpecName "kube-api-access-9rms6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:04:38.201571 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.201547 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e84e06-3163-4450-8cc7-278d239e6187-tls-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:04:38.201571 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.201570 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-home\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:04:38.201714 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.201579 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rms6\" (UniqueName: \"kubernetes.io/projected/d5e84e06-3163-4450-8cc7-278d239e6187-kube-api-access-9rms6\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:04:38.201714 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.201589 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-dshm\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:04:38.201714 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.201598 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e84e06-3163-4450-8cc7-278d239e6187-model-cache\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:04:38.301686 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.301658 2580 generic.go:356] "Generic (PLEG): container finished" podID="d5e84e06-3163-4450-8cc7-278d239e6187" containerID="0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4" exitCode=0 Sep 16 08:04:38.302029 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.301713 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" Sep 16 08:04:38.302029 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.301764 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" event={"ID":"d5e84e06-3163-4450-8cc7-278d239e6187","Type":"ContainerDied","Data":"0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4"} Sep 16 08:04:38.302029 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.301808 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg" event={"ID":"d5e84e06-3163-4450-8cc7-278d239e6187","Type":"ContainerDied","Data":"3f494ddd642dd2e84ec7f5aa616f3eca57ee2de0dadf5d4d36e78b7f5d0a0a50"} Sep 16 08:04:38.302029 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.301825 2580 scope.go:117] "RemoveContainer" containerID="0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4" Sep 16 08:04:38.312032 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.312013 2580 scope.go:117] "RemoveContainer" containerID="0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4" Sep 16 08:04:38.312332 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:04:38.312312 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4\": container with ID starting with 0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4 not found: ID does not exist" containerID="0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4" Sep 16 08:04:38.312385 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.312339 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4"} err="failed to get container status \"0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4\": rpc error: code = NotFound desc = could not find container \"0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4\": container with ID starting with 0357e71976d62e6845d25ef64a0f2d2b3cdd9ddebc497b16d68a76bfde8f24f4 not found: ID does not exist" Sep 16 08:04:38.324029 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.323982 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg"] Sep 16 08:04:38.326426 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.326405 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/rate-test-model-3616cc-kserve-6f7b895b4c-th5gg"] Sep 16 08:04:38.799340 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:38.799308 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:04:39.560653 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:39.560614 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e84e06-3163-4450-8cc7-278d239e6187" path="/var/lib/kubelet/pods/d5e84e06-3163-4450-8cc7-278d239e6187/volumes" Sep 16 08:04:59.130366 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.130335 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw"] Sep 16 08:04:59.130769 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.130755 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e84e06-3163-4450-8cc7-278d239e6187" containerName="main" Sep 16 08:04:59.130815 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.130770 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e84e06-3163-4450-8cc7-278d239e6187" containerName="main" Sep 16 08:04:59.130868 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.130850 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e84e06-3163-4450-8cc7-278d239e6187" containerName="main" Sep 16 08:04:59.134268 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.134245 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.136929 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.136897 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5493e8-serving-cert\"" Sep 16 08:04:59.145882 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.145857 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw"] Sep 16 08:04:59.277071 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.277042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.277270 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.277096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62531744-3fdb-4416-9e5c-6fc0254fa52f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.277270 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.277163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-metrics-certs\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.277270 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.277179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxq8\" (UniqueName: \"kubernetes.io/projected/62531744-3fdb-4416-9e5c-6fc0254fa52f-kube-api-access-tqxq8\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.378519 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.378479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-metrics-certs\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.378779 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.378521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxq8\" (UniqueName: \"kubernetes.io/projected/62531744-3fdb-4416-9e5c-6fc0254fa52f-kube-api-access-tqxq8\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.378779 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.378577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.378779 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.378646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62531744-3fdb-4416-9e5c-6fc0254fa52f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.378779 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:04:59.378765 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-serving-cert: secret "maas-api-e2e-ipp-b-5493e8-serving-cert" not found Sep 16 08:04:59.379040 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:04:59.378825 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-maas-api-tls podName:62531744-3fdb-4416-9e5c-6fc0254fa52f nodeName:}" failed. No retries permitted until 2026-09-16 08:04:59.878809092 +0000 UTC m=+1872.798425020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-maas-api-tls") pod "maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" (UID: "62531744-3fdb-4416-9e5c-6fc0254fa52f") : secret "maas-api-e2e-ipp-b-5493e8-serving-cert" not found Sep 16 08:04:59.379387 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.379365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62531744-3fdb-4416-9e5c-6fc0254fa52f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.381024 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.380971 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-metrics-certs\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.387683 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.387657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxq8\" (UniqueName: \"kubernetes.io/projected/62531744-3fdb-4416-9e5c-6fc0254fa52f-kube-api-access-tqxq8\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.884034 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.884002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:04:59.886722 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:04:59.886680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:05:00.048011 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:00.047965 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:05:00.321355 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:00.321327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw"] Sep 16 08:05:00.322637 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:05:00.322611 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62531744_3fdb_4416_9e5c_6fc0254fa52f.slice/crio-e856f52ed3d24fe3674ab3b89c087100ff2ad602006c6ffd42a274a3af2450ad WatchSource:0}: Error finding container e856f52ed3d24fe3674ab3b89c087100ff2ad602006c6ffd42a274a3af2450ad: Status 404 returned error can't find the container with id e856f52ed3d24fe3674ab3b89c087100ff2ad602006c6ffd42a274a3af2450ad Sep 16 08:05:00.438250 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:00.438193 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" event={"ID":"62531744-3fdb-4416-9e5c-6fc0254fa52f","Type":"ContainerStarted","Data":"e856f52ed3d24fe3674ab3b89c087100ff2ad602006c6ffd42a274a3af2450ad"} Sep 16 08:05:01.445356 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:01.445318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" event={"ID":"62531744-3fdb-4416-9e5c-6fc0254fa52f","Type":"ContainerStarted","Data":"edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af"} Sep 16 08:05:01.445766 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:01.445458 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:05:01.464032 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:01.463985 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" podStartSLOduration=1.99960659 podStartE2EDuration="2.463971473s" totalImagesPullingTime="464.364883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:05:01.462934051 +0000 UTC m=+1874.382549999" watchObservedRunningTime="2026-09-16 08:05:01.463971473 +0000 UTC m=+1874.383587418" Sep 16 08:05:04.314584 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:04.314539 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:05:07.457662 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:07.457634 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:05:26.909430 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:26.909397 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fh9w6_945b9e69-a7b4-458a-a638-5a9ffcc8cd8e/payload-processing/0.log" Sep 16 08:05:37.898442 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:37.898400 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:05:56.603242 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:05:56.603200 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:06:03.131199 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:03.131166 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fh9w6_945b9e69-a7b4-458a-a638-5a9ffcc8cd8e/payload-processing/0.log" Sep 16 08:06:09.832062 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:09.832028 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s"] Sep 16 08:06:09.837294 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:09.837270 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:09.840075 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:09.840047 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-d26a85-serving-cert\"" Sep 16 08:06:09.848541 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:09.848504 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s"] Sep 16 08:06:09.981678 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:09.981647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/508df6e7-6e39-40e3-834b-98f275f4914f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:09.981820 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:09.981717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:09.981820 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:09.981762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:09.981820 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:09.981789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5nz\" (UniqueName: \"kubernetes.io/projected/508df6e7-6e39-40e3-834b-98f275f4914f-kube-api-access-5h5nz\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.082813 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.082738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.082813 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.082796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.083028 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.082843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5nz\" (UniqueName: \"kubernetes.io/projected/508df6e7-6e39-40e3-834b-98f275f4914f-kube-api-access-5h5nz\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.083028 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.082892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/508df6e7-6e39-40e3-834b-98f275f4914f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.083574 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.083548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/508df6e7-6e39-40e3-834b-98f275f4914f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.085222 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.085196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.085313 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.085258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.091073 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.091048 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5nz\" (UniqueName: \"kubernetes.io/projected/508df6e7-6e39-40e3-834b-98f275f4914f-kube-api-access-5h5nz\") pod \"maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.149148 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.149113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.426821 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.426746 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s"] Sep 16 08:06:10.427671 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:06:10.427637 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508df6e7_6e39_40e3_834b_98f275f4914f.slice/crio-3d7b2d1ed69564dbc7e72a97c2ba38799f0f8e65f58e7ed4bbc96de278d5e1c9 WatchSource:0}: Error finding container 3d7b2d1ed69564dbc7e72a97c2ba38799f0f8e65f58e7ed4bbc96de278d5e1c9: Status 404 returned error can't find the container with id 3d7b2d1ed69564dbc7e72a97c2ba38799f0f8e65f58e7ed4bbc96de278d5e1c9 Sep 16 08:06:10.853956 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.853921 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" event={"ID":"508df6e7-6e39-40e3-834b-98f275f4914f","Type":"ContainerStarted","Data":"c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939"} Sep 16 08:06:10.854355 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.854015 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:10.854355 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.854036 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" event={"ID":"508df6e7-6e39-40e3-834b-98f275f4914f","Type":"ContainerStarted","Data":"3d7b2d1ed69564dbc7e72a97c2ba38799f0f8e65f58e7ed4bbc96de278d5e1c9"} Sep 16 08:06:10.873608 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:10.873565 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" podStartSLOduration=1.537229261 podStartE2EDuration="1.873549846s" totalImagesPullingTime="336.320585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:06:10.871295452 +0000 UTC m=+1943.790911402" watchObservedRunningTime="2026-09-16 08:06:10.873549846 +0000 UTC m=+1943.793165832" Sep 16 08:06:16.865586 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:16.865555 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:17.005611 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:17.005569 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:06:39.478988 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.478950 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s"] Sep 16 08:06:39.479543 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.479253 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" podUID="508df6e7-6e39-40e3-834b-98f275f4914f" containerName="maas-api" containerID="cri-o://c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939" gracePeriod=30 Sep 16 08:06:39.723535 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.723509 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:39.839755 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.839673 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h5nz\" (UniqueName: \"kubernetes.io/projected/508df6e7-6e39-40e3-834b-98f275f4914f-kube-api-access-5h5nz\") pod \"508df6e7-6e39-40e3-834b-98f275f4914f\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " Sep 16 08:06:39.839755 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.839713 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/508df6e7-6e39-40e3-834b-98f275f4914f-openshift-service-ca\") pod \"508df6e7-6e39-40e3-834b-98f275f4914f\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " Sep 16 08:06:39.839755 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.839756 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-maas-api-tls\") pod \"508df6e7-6e39-40e3-834b-98f275f4914f\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " Sep 16 08:06:39.840038 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.839795 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-metrics-certs\") pod \"508df6e7-6e39-40e3-834b-98f275f4914f\" (UID: \"508df6e7-6e39-40e3-834b-98f275f4914f\") " Sep 16 08:06:39.840144 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.840113 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508df6e7-6e39-40e3-834b-98f275f4914f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "508df6e7-6e39-40e3-834b-98f275f4914f" (UID: "508df6e7-6e39-40e3-834b-98f275f4914f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:06:39.841899 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.841866 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "508df6e7-6e39-40e3-834b-98f275f4914f" (UID: "508df6e7-6e39-40e3-834b-98f275f4914f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:06:39.842026 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.841923 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508df6e7-6e39-40e3-834b-98f275f4914f-kube-api-access-5h5nz" (OuterVolumeSpecName: "kube-api-access-5h5nz") pod "508df6e7-6e39-40e3-834b-98f275f4914f" (UID: "508df6e7-6e39-40e3-834b-98f275f4914f"). InnerVolumeSpecName "kube-api-access-5h5nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:06:39.842026 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.841979 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "508df6e7-6e39-40e3-834b-98f275f4914f" (UID: "508df6e7-6e39-40e3-834b-98f275f4914f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:06:39.941315 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.941281 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:06:39.941315 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.941307 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/508df6e7-6e39-40e3-834b-98f275f4914f-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:06:39.941315 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.941317 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h5nz\" (UniqueName: \"kubernetes.io/projected/508df6e7-6e39-40e3-834b-98f275f4914f-kube-api-access-5h5nz\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:06:39.941524 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:39.941326 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/508df6e7-6e39-40e3-834b-98f275f4914f-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:06:40.021461 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:40.021388 2580 generic.go:356] "Generic (PLEG): container finished" podID="508df6e7-6e39-40e3-834b-98f275f4914f" containerID="c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939" exitCode=0 Sep 16 08:06:40.021461 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:40.021436 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" Sep 16 08:06:40.021729 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:40.021483 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" event={"ID":"508df6e7-6e39-40e3-834b-98f275f4914f","Type":"ContainerDied","Data":"c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939"} Sep 16 08:06:40.021729 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:40.021547 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s" event={"ID":"508df6e7-6e39-40e3-834b-98f275f4914f","Type":"ContainerDied","Data":"3d7b2d1ed69564dbc7e72a97c2ba38799f0f8e65f58e7ed4bbc96de278d5e1c9"} Sep 16 08:06:40.021729 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:40.021580 2580 scope.go:117] "RemoveContainer" containerID="c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939" Sep 16 08:06:40.032097 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:40.032080 2580 scope.go:117] "RemoveContainer" containerID="c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939" Sep 16 08:06:40.032394 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:06:40.032376 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939\": container with ID starting with c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939 not found: ID does not exist" containerID="c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939" Sep 16 08:06:40.032448 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:40.032403 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939"} err="failed to get container status \"c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939\": rpc error: code = NotFound desc = could not find container \"c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939\": container with ID starting with c835cb9fefe1a42e075adf12deb1405443ed3fb43247713ab48b2b3d6dfe1939 not found: ID does not exist" Sep 16 08:06:40.058899 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:40.058864 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s"] Sep 16 08:06:40.076527 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:40.076505 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d26a85-5b4d4d49c-f9h8s"] Sep 16 08:06:41.560661 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:41.560629 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="508df6e7-6e39-40e3-834b-98f275f4914f" path="/var/lib/kubelet/pods/508df6e7-6e39-40e3-834b-98f275f4914f/volumes" Sep 16 08:06:44.103200 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:06:44.103166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:07:03.510598 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:03.510560 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:07:05.669057 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.669012 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq"] Sep 16 08:07:05.669535 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.669293 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" podUID="ccea580a-db9b-46a5-965c-fe7421b4ee00" containerName="maas-api" containerID="cri-o://bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e" gracePeriod=30 Sep 16 08:07:05.927827 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.927771 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:07:05.971572 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.971539 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pjsj\" (UniqueName: \"kubernetes.io/projected/ccea580a-db9b-46a5-965c-fe7421b4ee00-kube-api-access-7pjsj\") pod \"ccea580a-db9b-46a5-965c-fe7421b4ee00\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " Sep 16 08:07:05.971732 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.971608 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-maas-api-tls\") pod \"ccea580a-db9b-46a5-965c-fe7421b4ee00\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " Sep 16 08:07:05.971732 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.971690 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccea580a-db9b-46a5-965c-fe7421b4ee00-openshift-service-ca\") pod \"ccea580a-db9b-46a5-965c-fe7421b4ee00\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " Sep 16 08:07:05.971732 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.971714 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-metrics-certs\") pod \"ccea580a-db9b-46a5-965c-fe7421b4ee00\" (UID: \"ccea580a-db9b-46a5-965c-fe7421b4ee00\") " Sep 16 08:07:05.972027 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.972003 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccea580a-db9b-46a5-965c-fe7421b4ee00-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ccea580a-db9b-46a5-965c-fe7421b4ee00" (UID: "ccea580a-db9b-46a5-965c-fe7421b4ee00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:07:05.973753 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.973725 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccea580a-db9b-46a5-965c-fe7421b4ee00-kube-api-access-7pjsj" (OuterVolumeSpecName: "kube-api-access-7pjsj") pod "ccea580a-db9b-46a5-965c-fe7421b4ee00" (UID: "ccea580a-db9b-46a5-965c-fe7421b4ee00"). InnerVolumeSpecName "kube-api-access-7pjsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:07:05.973854 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.973727 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ccea580a-db9b-46a5-965c-fe7421b4ee00" (UID: "ccea580a-db9b-46a5-965c-fe7421b4ee00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:07:05.973854 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:05.973834 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ccea580a-db9b-46a5-965c-fe7421b4ee00" (UID: "ccea580a-db9b-46a5-965c-fe7421b4ee00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:07:06.072448 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.072416 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pjsj\" (UniqueName: \"kubernetes.io/projected/ccea580a-db9b-46a5-965c-fe7421b4ee00-kube-api-access-7pjsj\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:07:06.072448 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.072445 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:07:06.072448 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.072455 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccea580a-db9b-46a5-965c-fe7421b4ee00-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:07:06.072704 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.072463 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccea580a-db9b-46a5-965c-fe7421b4ee00-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:07:06.175775 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.175742 2580 generic.go:356] "Generic (PLEG): container finished" podID="ccea580a-db9b-46a5-965c-fe7421b4ee00" containerID="bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e" exitCode=0 Sep 16 08:07:06.175938 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.175783 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" Sep 16 08:07:06.175938 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.175827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" event={"ID":"ccea580a-db9b-46a5-965c-fe7421b4ee00","Type":"ContainerDied","Data":"bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e"} Sep 16 08:07:06.175938 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.175879 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq" event={"ID":"ccea580a-db9b-46a5-965c-fe7421b4ee00","Type":"ContainerDied","Data":"48d7636a16e351cc2ee2a61de96217a2063394408f40a3eaba8f5c0a3bb016a9"} Sep 16 08:07:06.175938 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.175902 2580 scope.go:117] "RemoveContainer" containerID="bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e" Sep 16 08:07:06.189553 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.189533 2580 scope.go:117] "RemoveContainer" containerID="bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e" Sep 16 08:07:06.189863 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:07:06.189838 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e\": container with ID starting with bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e not found: ID does not exist" containerID="bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e" Sep 16 08:07:06.189952 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.189878 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e"} err="failed to get container status \"bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e\": rpc error: code = NotFound desc = could not find container \"bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e\": container with ID starting with bd691d935856dedea2b92cccae7828988376f7cdd0c5233c94bd6e46609a925e not found: ID does not exist" Sep 16 08:07:06.200994 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.200971 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq"] Sep 16 08:07:06.207859 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:06.207836 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-796e11-79bf7975d7-dhpmq"] Sep 16 08:07:07.561717 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:07.561683 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccea580a-db9b-46a5-965c-fe7421b4ee00" path="/var/lib/kubelet/pods/ccea580a-db9b-46a5-965c-fe7421b4ee00/volumes" Sep 16 08:07:17.103363 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:07:17.103326 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:08:04.889872 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:04.889835 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw"] Sep 16 08:08:04.890370 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:04.890055 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" podUID="62531744-3fdb-4416-9e5c-6fc0254fa52f" containerName="maas-api" containerID="cri-o://edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af" gracePeriod=30 Sep 16 08:08:05.149686 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.149627 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:08:05.176969 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.176942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-maas-api-tls\") pod \"62531744-3fdb-4416-9e5c-6fc0254fa52f\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " Sep 16 08:08:05.177101 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.177020 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqxq8\" (UniqueName: \"kubernetes.io/projected/62531744-3fdb-4416-9e5c-6fc0254fa52f-kube-api-access-tqxq8\") pod \"62531744-3fdb-4416-9e5c-6fc0254fa52f\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " Sep 16 08:08:05.177101 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.177055 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-metrics-certs\") pod \"62531744-3fdb-4416-9e5c-6fc0254fa52f\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " Sep 16 08:08:05.177101 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.177076 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62531744-3fdb-4416-9e5c-6fc0254fa52f-openshift-service-ca\") pod \"62531744-3fdb-4416-9e5c-6fc0254fa52f\" (UID: \"62531744-3fdb-4416-9e5c-6fc0254fa52f\") " Sep 16 08:08:05.177571 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.177541 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62531744-3fdb-4416-9e5c-6fc0254fa52f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62531744-3fdb-4416-9e5c-6fc0254fa52f" (UID: "62531744-3fdb-4416-9e5c-6fc0254fa52f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:08:05.179269 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.179185 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62531744-3fdb-4416-9e5c-6fc0254fa52f" (UID: "62531744-3fdb-4416-9e5c-6fc0254fa52f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:08:05.179703 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.179677 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "62531744-3fdb-4416-9e5c-6fc0254fa52f" (UID: "62531744-3fdb-4416-9e5c-6fc0254fa52f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:08:05.179703 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.179693 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62531744-3fdb-4416-9e5c-6fc0254fa52f-kube-api-access-tqxq8" (OuterVolumeSpecName: "kube-api-access-tqxq8") pod "62531744-3fdb-4416-9e5c-6fc0254fa52f" (UID: "62531744-3fdb-4416-9e5c-6fc0254fa52f"). InnerVolumeSpecName "kube-api-access-tqxq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:05.277964 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.277934 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxq8\" (UniqueName: \"kubernetes.io/projected/62531744-3fdb-4416-9e5c-6fc0254fa52f-kube-api-access-tqxq8\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:05.277964 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.277962 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:05.278147 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.277977 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62531744-3fdb-4416-9e5c-6fc0254fa52f-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:05.278147 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.277989 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62531744-3fdb-4416-9e5c-6fc0254fa52f-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:05.517456 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.517421 2580 generic.go:356] "Generic (PLEG): container finished" podID="62531744-3fdb-4416-9e5c-6fc0254fa52f" containerID="edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af" exitCode=0 Sep 16 08:08:05.517621 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.517464 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" event={"ID":"62531744-3fdb-4416-9e5c-6fc0254fa52f","Type":"ContainerDied","Data":"edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af"} Sep 16 08:08:05.517621 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.517504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" event={"ID":"62531744-3fdb-4416-9e5c-6fc0254fa52f","Type":"ContainerDied","Data":"e856f52ed3d24fe3674ab3b89c087100ff2ad602006c6ffd42a274a3af2450ad"} Sep 16 08:08:05.517621 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.517524 2580 scope.go:117] "RemoveContainer" containerID="edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af" Sep 16 08:08:05.517621 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.517469 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw" Sep 16 08:08:05.527572 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.527549 2580 scope.go:117] "RemoveContainer" containerID="edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af" Sep 16 08:08:05.527854 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:08:05.527828 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af\": container with ID starting with edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af not found: ID does not exist" containerID="edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af" Sep 16 08:08:05.527955 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.527859 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af"} err="failed to get container status \"edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af\": rpc error: code = NotFound desc = could not find container \"edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af\": container with ID starting with edcbb23d14e8f87604d33fdb5b8c85b56cdc2eedca949eacb39450b75e0d65af not found: ID does not exist" Sep 16 08:08:05.539898 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.539875 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw"] Sep 16 08:08:05.545846 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.545821 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5493e8-596c5d6bcf-c95hw"] Sep 16 08:08:05.560461 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:05.560440 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62531744-3fdb-4416-9e5c-6fc0254fa52f" path="/var/lib/kubelet/pods/62531744-3fdb-4416-9e5c-6fc0254fa52f/volumes" Sep 16 08:08:08.197059 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:08.197020 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:08:22.225742 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.225665 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2"] Sep 16 08:08:22.226336 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.225879 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" podUID="ffc8796c-0eef-4ea7-b618-270cb14a6401" containerName="maas-api" containerID="cri-o://fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e" gracePeriod=30 Sep 16 08:08:22.482497 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.482440 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 08:08:22.622747 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.622715 2580 generic.go:356] "Generic (PLEG): container finished" podID="ffc8796c-0eef-4ea7-b618-270cb14a6401" containerID="fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e" exitCode=0 Sep 16 08:08:22.622924 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.622755 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" Sep 16 08:08:22.622924 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.622787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" event={"ID":"ffc8796c-0eef-4ea7-b618-270cb14a6401","Type":"ContainerDied","Data":"fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e"} Sep 16 08:08:22.622924 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.622842 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2" event={"ID":"ffc8796c-0eef-4ea7-b618-270cb14a6401","Type":"ContainerDied","Data":"27697abe27fdd566a55548573ebdf6022316822c384e83627430b4401590899e"} Sep 16 08:08:22.622924 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.622869 2580 scope.go:117] "RemoveContainer" containerID="fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e" Sep 16 08:08:22.623450 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.623432 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kx68\" (UniqueName: \"kubernetes.io/projected/ffc8796c-0eef-4ea7-b618-270cb14a6401-kube-api-access-2kx68\") pod \"ffc8796c-0eef-4ea7-b618-270cb14a6401\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " Sep 16 08:08:22.623567 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.623485 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-metrics-certs\") pod \"ffc8796c-0eef-4ea7-b618-270cb14a6401\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " Sep 16 08:08:22.623567 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.623519 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc8796c-0eef-4ea7-b618-270cb14a6401-openshift-service-ca\") pod \"ffc8796c-0eef-4ea7-b618-270cb14a6401\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " Sep 16 08:08:22.623839 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.623721 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-maas-api-tls\") pod \"ffc8796c-0eef-4ea7-b618-270cb14a6401\" (UID: \"ffc8796c-0eef-4ea7-b618-270cb14a6401\") " Sep 16 08:08:22.623983 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.623899 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc8796c-0eef-4ea7-b618-270cb14a6401-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffc8796c-0eef-4ea7-b618-270cb14a6401" (UID: "ffc8796c-0eef-4ea7-b618-270cb14a6401"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:08:22.624201 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.624180 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc8796c-0eef-4ea7-b618-270cb14a6401-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:22.626244 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.626195 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ffc8796c-0eef-4ea7-b618-270cb14a6401" (UID: "ffc8796c-0eef-4ea7-b618-270cb14a6401"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:08:22.626371 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.626301 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc8796c-0eef-4ea7-b618-270cb14a6401-kube-api-access-2kx68" (OuterVolumeSpecName: "kube-api-access-2kx68") pod "ffc8796c-0eef-4ea7-b618-270cb14a6401" (UID: "ffc8796c-0eef-4ea7-b618-270cb14a6401"). InnerVolumeSpecName "kube-api-access-2kx68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:22.626552 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.626524 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffc8796c-0eef-4ea7-b618-270cb14a6401" (UID: "ffc8796c-0eef-4ea7-b618-270cb14a6401"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:08:22.644992 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.644975 2580 scope.go:117] "RemoveContainer" containerID="fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e" Sep 16 08:08:22.645290 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:08:22.645267 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e\": container with ID starting with fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e not found: ID does not exist" containerID="fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e" Sep 16 08:08:22.645387 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.645298 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e"} err="failed to get container status \"fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e\": rpc error: code = NotFound desc = could not find container \"fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e\": container with ID starting with fdd9fbc62054a78d8a9d240b1b3ca15f71d553d4ba0ca77352bbb3eac8c11d3e not found: ID does not exist" Sep 16 08:08:22.725168 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.725143 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:22.725168 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.725167 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kx68\" (UniqueName: \"kubernetes.io/projected/ffc8796c-0eef-4ea7-b618-270cb14a6401-kube-api-access-2kx68\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:22.725349 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.725176 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc8796c-0eef-4ea7-b618-270cb14a6401-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:22.950963 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.950932 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2"] Sep 16 08:08:22.959536 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:22.959508 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3616cc-d48d564ff-zf5w2"] Sep 16 08:08:23.561348 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:23.561311 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc8796c-0eef-4ea7-b618-270cb14a6401" path="/var/lib/kubelet/pods/ffc8796c-0eef-4ea7-b618-270cb14a6401/volumes" Sep 16 08:08:25.204059 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:25.204027 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:08:39.843043 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:39.843010 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n"] Sep 16 08:08:39.843519 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:39.843350 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" podUID="9f301562-0f6f-4c51-bfce-bf0a2e8b405c" containerName="maas-api" containerID="cri-o://1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3" gracePeriod=30 Sep 16 08:08:40.095645 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.095596 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 08:08:40.269924 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.269885 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-maas-api-tls\") pod \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " Sep 16 08:08:40.269924 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.269929 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmls6\" (UniqueName: \"kubernetes.io/projected/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-kube-api-access-rmls6\") pod \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " Sep 16 08:08:40.270175 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.270023 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-metrics-certs\") pod \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " Sep 16 08:08:40.270175 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.270054 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-openshift-service-ca\") pod \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\" (UID: \"9f301562-0f6f-4c51-bfce-bf0a2e8b405c\") " Sep 16 08:08:40.270532 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.270500 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f301562-0f6f-4c51-bfce-bf0a2e8b405c" (UID: "9f301562-0f6f-4c51-bfce-bf0a2e8b405c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:08:40.272236 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.272196 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9f301562-0f6f-4c51-bfce-bf0a2e8b405c" (UID: "9f301562-0f6f-4c51-bfce-bf0a2e8b405c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:08:40.272333 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.272227 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f301562-0f6f-4c51-bfce-bf0a2e8b405c" (UID: "9f301562-0f6f-4c51-bfce-bf0a2e8b405c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:08:40.272333 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.272238 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-kube-api-access-rmls6" (OuterVolumeSpecName: "kube-api-access-rmls6") pod "9f301562-0f6f-4c51-bfce-bf0a2e8b405c" (UID: "9f301562-0f6f-4c51-bfce-bf0a2e8b405c"). InnerVolumeSpecName "kube-api-access-rmls6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:40.371688 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.371626 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:40.371688 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.371650 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:40.371688 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.371662 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:40.371688 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.371670 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmls6\" (UniqueName: \"kubernetes.io/projected/9f301562-0f6f-4c51-bfce-bf0a2e8b405c-kube-api-access-rmls6\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:40.723387 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.723351 2580 generic.go:356] "Generic (PLEG): container finished" podID="9f301562-0f6f-4c51-bfce-bf0a2e8b405c" containerID="1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3" exitCode=0 Sep 16 08:08:40.723537 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.723395 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" Sep 16 08:08:40.723537 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.723434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" event={"ID":"9f301562-0f6f-4c51-bfce-bf0a2e8b405c","Type":"ContainerDied","Data":"1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3"} Sep 16 08:08:40.723537 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.723481 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n" event={"ID":"9f301562-0f6f-4c51-bfce-bf0a2e8b405c","Type":"ContainerDied","Data":"5760ff2f2510f246e0c2ca4cacbaaf84cf681668479f0fce16404ec9f49f0fa7"} Sep 16 08:08:40.723537 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.723502 2580 scope.go:117] "RemoveContainer" containerID="1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3" Sep 16 08:08:40.732997 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.732981 2580 scope.go:117] "RemoveContainer" containerID="1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3" Sep 16 08:08:40.733287 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:08:40.733270 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3\": container with ID starting with 1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3 not found: ID does not exist" containerID="1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3" Sep 16 08:08:40.733353 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.733292 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3"} err="failed to get container status \"1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3\": rpc error: code = NotFound desc = could not find container \"1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3\": container with ID starting with 1d2eb8dedc617e887584c9cd65e3e7e68dd6e7402895df077b85a86d036718a3 not found: ID does not exist" Sep 16 08:08:40.747841 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.747814 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n"] Sep 16 08:08:40.751644 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:40.751625 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e9fc8c-5cbfbb6bf6-w848n"] Sep 16 08:08:41.560765 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:41.560723 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f301562-0f6f-4c51-bfce-bf0a2e8b405c" path="/var/lib/kubelet/pods/9f301562-0f6f-4c51-bfce-bf0a2e8b405c/volumes" Sep 16 08:08:42.700671 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:42.700631 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:08:47.683346 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:47.683311 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 08:08:47.691967 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:47.691945 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 08:08:49.371247 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.371189 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-5tt6v"] Sep 16 08:08:49.371735 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.371474 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" podUID="68cccfa2-901c-4ff8-a8c5-f4af1ad249a9" containerName="manager" containerID="cri-o://6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753" gracePeriod=30 Sep 16 08:08:49.726934 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.726912 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 08:08:49.750334 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.749452 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-metrics-certs\") pod \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " Sep 16 08:08:49.750783 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.750507 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvms7\" (UniqueName: \"kubernetes.io/projected/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-kube-api-access-qvms7\") pod \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " Sep 16 08:08:49.750783 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.750552 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-webhook-cert\") pod \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\" (UID: \"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9\") " Sep 16 08:08:49.753815 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.753771 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-kube-api-access-qvms7" (OuterVolumeSpecName: "kube-api-access-qvms7") pod "68cccfa2-901c-4ff8-a8c5-f4af1ad249a9" (UID: "68cccfa2-901c-4ff8-a8c5-f4af1ad249a9"). InnerVolumeSpecName "kube-api-access-qvms7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:49.754019 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.753996 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "68cccfa2-901c-4ff8-a8c5-f4af1ad249a9" (UID: "68cccfa2-901c-4ff8-a8c5-f4af1ad249a9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:08:49.754295 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.754268 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68cccfa2-901c-4ff8-a8c5-f4af1ad249a9" (UID: "68cccfa2-901c-4ff8-a8c5-f4af1ad249a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:08:49.776967 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.776940 2580 generic.go:356] "Generic (PLEG): container finished" podID="68cccfa2-901c-4ff8-a8c5-f4af1ad249a9" containerID="6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753" exitCode=0 Sep 16 08:08:49.777094 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.776984 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" Sep 16 08:08:49.777094 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.777066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" event={"ID":"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9","Type":"ContainerDied","Data":"6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753"} Sep 16 08:08:49.777234 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.777128 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd9d7d98-5tt6v" event={"ID":"68cccfa2-901c-4ff8-a8c5-f4af1ad249a9","Type":"ContainerDied","Data":"1f02c3aef56c44da3a7ba2667ef9e8532c341e1fb64d8ff3dfd8b56e4d0639f4"} Sep 16 08:08:49.777234 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.777167 2580 scope.go:117] "RemoveContainer" containerID="6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753" Sep 16 08:08:49.788336 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.788319 2580 scope.go:117] "RemoveContainer" containerID="6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753" Sep 16 08:08:49.788607 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:08:49.788592 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753\": container with ID starting with 6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753 not found: ID does not exist" containerID="6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753" Sep 16 08:08:49.788662 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.788618 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753"} err="failed to get container status \"6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753\": rpc error: code = NotFound desc = could not find container \"6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753\": container with ID starting with 6272c9a357c464fd25045ee4a888156fd21a0192b87b907a07c79c2894368753 not found: ID does not exist" Sep 16 08:08:49.800161 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.800141 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-5tt6v"] Sep 16 08:08:49.804182 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.804163 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-5tt6v"] Sep 16 08:08:49.851686 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.851666 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvms7\" (UniqueName: \"kubernetes.io/projected/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-kube-api-access-qvms7\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:49.851686 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.851687 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-webhook-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:49.851816 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:49.851696 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:08:51.562782 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:51.562741 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68cccfa2-901c-4ff8-a8c5-f4af1ad249a9" path="/var/lib/kubelet/pods/68cccfa2-901c-4ff8-a8c5-f4af1ad249a9/volumes" Sep 16 08:08:51.613758 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:08:51.613714 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:09:50.631771 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.631733 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-kjwd9"] Sep 16 08:09:50.632314 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632295 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc8796c-0eef-4ea7-b618-270cb14a6401" containerName="maas-api" Sep 16 08:09:50.632404 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632317 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc8796c-0eef-4ea7-b618-270cb14a6401" containerName="maas-api" Sep 16 08:09:50.632404 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632330 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62531744-3fdb-4416-9e5c-6fc0254fa52f" containerName="maas-api" Sep 16 08:09:50.632404 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632339 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62531744-3fdb-4416-9e5c-6fc0254fa52f" containerName="maas-api" Sep 16 08:09:50.632404 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632366 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68cccfa2-901c-4ff8-a8c5-f4af1ad249a9" containerName="manager" Sep 16 08:09:50.632404 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632375 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="68cccfa2-901c-4ff8-a8c5-f4af1ad249a9" containerName="manager" Sep 16 08:09:50.632404 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632399 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508df6e7-6e39-40e3-834b-98f275f4914f" containerName="maas-api" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632407 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="508df6e7-6e39-40e3-834b-98f275f4914f" containerName="maas-api" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632423 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f301562-0f6f-4c51-bfce-bf0a2e8b405c" containerName="maas-api" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632432 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f301562-0f6f-4c51-bfce-bf0a2e8b405c" containerName="maas-api" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632445 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccea580a-db9b-46a5-965c-fe7421b4ee00" containerName="maas-api" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632454 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccea580a-db9b-46a5-965c-fe7421b4ee00" containerName="maas-api" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632558 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="508df6e7-6e39-40e3-834b-98f275f4914f" containerName="maas-api" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632570 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccea580a-db9b-46a5-965c-fe7421b4ee00" containerName="maas-api" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632580 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62531744-3fdb-4416-9e5c-6fc0254fa52f" containerName="maas-api" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632589 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc8796c-0eef-4ea7-b618-270cb14a6401" containerName="maas-api" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632601 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="68cccfa2-901c-4ff8-a8c5-f4af1ad249a9" containerName="manager" Sep 16 08:09:50.632705 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.632609 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f301562-0f6f-4c51-bfce-bf0a2e8b405c" containerName="maas-api" Sep 16 08:09:50.636303 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.636281 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:50.639313 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.639289 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 08:09:50.641094 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.640849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 08:09:50.641094 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.640954 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nvkmk\"" Sep 16 08:09:50.641290 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.640954 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 08:09:50.644924 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.644905 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-kjwd9"] Sep 16 08:09:50.728082 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.728050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-metrics-certs\") pod \"maas-controller-bbd9d7d98-kjwd9\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:50.728193 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.728093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-webhook-cert\") pod \"maas-controller-bbd9d7d98-kjwd9\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:50.728193 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.728163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-656pj\" (UniqueName: \"kubernetes.io/projected/7163e1f0-f850-4dfd-8342-dfe9df02bec1-kube-api-access-656pj\") pod \"maas-controller-bbd9d7d98-kjwd9\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:50.828545 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.828519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-metrics-certs\") pod \"maas-controller-bbd9d7d98-kjwd9\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:50.828644 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.828555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-webhook-cert\") pod \"maas-controller-bbd9d7d98-kjwd9\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:50.828644 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.828602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-656pj\" (UniqueName: \"kubernetes.io/projected/7163e1f0-f850-4dfd-8342-dfe9df02bec1-kube-api-access-656pj\") pod \"maas-controller-bbd9d7d98-kjwd9\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:50.830926 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.830897 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-webhook-cert\") pod \"maas-controller-bbd9d7d98-kjwd9\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:50.831019 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.830927 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-metrics-certs\") pod \"maas-controller-bbd9d7d98-kjwd9\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:50.836721 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.836701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-656pj\" (UniqueName: \"kubernetes.io/projected/7163e1f0-f850-4dfd-8342-dfe9df02bec1-kube-api-access-656pj\") pod \"maas-controller-bbd9d7d98-kjwd9\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:50.948541 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:50.948513 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:51.219297 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:51.219272 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-kjwd9"] Sep 16 08:09:51.220702 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:09:51.220673 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7163e1f0_f850_4dfd_8342_dfe9df02bec1.slice/crio-58876568a8bb49370a0960fd8cf91b698fd397926b9f41c11e76bed87353ce16 WatchSource:0}: Error finding container 58876568a8bb49370a0960fd8cf91b698fd397926b9f41c11e76bed87353ce16: Status 404 returned error can't find the container with id 58876568a8bb49370a0960fd8cf91b698fd397926b9f41c11e76bed87353ce16 Sep 16 08:09:51.222062 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:51.222044 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 08:09:52.127608 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:52.127520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" event={"ID":"7163e1f0-f850-4dfd-8342-dfe9df02bec1","Type":"ContainerStarted","Data":"2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2"} Sep 16 08:09:52.127608 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:52.127572 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" event={"ID":"7163e1f0-f850-4dfd-8342-dfe9df02bec1","Type":"ContainerStarted","Data":"58876568a8bb49370a0960fd8cf91b698fd397926b9f41c11e76bed87353ce16"} Sep 16 08:09:52.128016 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:52.127624 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:09:52.146508 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:09:52.146462 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" podStartSLOduration=1.651281338 podStartE2EDuration="2.14644571s" totalImagesPullingTime="495.164372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:09:52.145409119 +0000 UTC m=+2165.065025067" watchObservedRunningTime="2026-09-16 08:09:52.14644571 +0000 UTC m=+2165.066061657" Sep 16 08:10:03.136034 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:03.136001 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:10:12.603887 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:12.603847 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-kjwd9"] Sep 16 08:10:12.604358 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:12.604060 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" podUID="7163e1f0-f850-4dfd-8342-dfe9df02bec1" containerName="manager" containerID="cri-o://2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2" gracePeriod=30 Sep 16 08:10:12.854204 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:12.854149 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:10:12.905535 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:12.905507 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-webhook-cert\") pod \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " Sep 16 08:10:12.905691 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:12.905559 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-656pj\" (UniqueName: \"kubernetes.io/projected/7163e1f0-f850-4dfd-8342-dfe9df02bec1-kube-api-access-656pj\") pod \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " Sep 16 08:10:12.905691 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:12.905612 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-metrics-certs\") pod \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\" (UID: \"7163e1f0-f850-4dfd-8342-dfe9df02bec1\") " Sep 16 08:10:12.907721 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:12.907689 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7163e1f0-f850-4dfd-8342-dfe9df02bec1" (UID: "7163e1f0-f850-4dfd-8342-dfe9df02bec1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:10:12.907899 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:12.907714 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7163e1f0-f850-4dfd-8342-dfe9df02bec1" (UID: "7163e1f0-f850-4dfd-8342-dfe9df02bec1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:10:12.907899 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:12.907699 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7163e1f0-f850-4dfd-8342-dfe9df02bec1-kube-api-access-656pj" (OuterVolumeSpecName: "kube-api-access-656pj") pod "7163e1f0-f850-4dfd-8342-dfe9df02bec1" (UID: "7163e1f0-f850-4dfd-8342-dfe9df02bec1"). InnerVolumeSpecName "kube-api-access-656pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:10:13.006761 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.006729 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-webhook-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:10:13.006761 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.006762 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-656pj\" (UniqueName: \"kubernetes.io/projected/7163e1f0-f850-4dfd-8342-dfe9df02bec1-kube-api-access-656pj\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:10:13.006963 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.006776 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7163e1f0-f850-4dfd-8342-dfe9df02bec1-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:10:13.112138 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.112052 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:10:13.246302 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.246265 2580 generic.go:356] "Generic (PLEG): container finished" podID="7163e1f0-f850-4dfd-8342-dfe9df02bec1" containerID="2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2" exitCode=0 Sep 16 08:10:13.246434 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.246311 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" Sep 16 08:10:13.246434 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.246354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" event={"ID":"7163e1f0-f850-4dfd-8342-dfe9df02bec1","Type":"ContainerDied","Data":"2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2"} Sep 16 08:10:13.246434 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.246406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd9d7d98-kjwd9" event={"ID":"7163e1f0-f850-4dfd-8342-dfe9df02bec1","Type":"ContainerDied","Data":"58876568a8bb49370a0960fd8cf91b698fd397926b9f41c11e76bed87353ce16"} Sep 16 08:10:13.246434 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.246433 2580 scope.go:117] "RemoveContainer" containerID="2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2" Sep 16 08:10:13.256195 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.256177 2580 scope.go:117] "RemoveContainer" containerID="2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2" Sep 16 08:10:13.256475 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:10:13.256459 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2\": container with ID starting with 2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2 not found: ID does not exist" containerID="2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2" Sep 16 08:10:13.256521 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.256483 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2"} err="failed to get container status \"2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2\": rpc error: code = NotFound desc = could not find container \"2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2\": container with ID starting with 2f37427c13ec9c82a666db04b540fdfe9666aa96f3a75d28a0c761ed8281b7f2 not found: ID does not exist" Sep 16 08:10:13.268627 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.268602 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-kjwd9"] Sep 16 08:10:13.273877 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.273854 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-kjwd9"] Sep 16 08:10:13.560318 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:13.560290 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7163e1f0-f850-4dfd-8342-dfe9df02bec1" path="/var/lib/kubelet/pods/7163e1f0-f850-4dfd-8342-dfe9df02bec1/volumes" Sep 16 08:10:14.112968 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.112935 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-spcd9"] Sep 16 08:10:14.116284 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.113920 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7163e1f0-f850-4dfd-8342-dfe9df02bec1" containerName="manager" Sep 16 08:10:14.116284 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.113952 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7163e1f0-f850-4dfd-8342-dfe9df02bec1" containerName="manager" Sep 16 08:10:14.116284 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.114252 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7163e1f0-f850-4dfd-8342-dfe9df02bec1" containerName="manager" Sep 16 08:10:14.120580 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.119977 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.125079 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.125057 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 08:10:14.126141 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.126122 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 08:10:14.126545 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.126527 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 08:10:14.126903 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.126881 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nvkmk\"" Sep 16 08:10:14.129780 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.129356 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-spcd9"] Sep 16 08:10:14.222641 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.222614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9814843d-9c41-4f9c-b91d-adcd61ed81c2-webhook-cert\") pod \"maas-controller-bbd9d7d98-spcd9\" (UID: \"9814843d-9c41-4f9c-b91d-adcd61ed81c2\") " pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.222760 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.222669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6fs\" (UniqueName: \"kubernetes.io/projected/9814843d-9c41-4f9c-b91d-adcd61ed81c2-kube-api-access-rb6fs\") pod \"maas-controller-bbd9d7d98-spcd9\" (UID: \"9814843d-9c41-4f9c-b91d-adcd61ed81c2\") " pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.222760 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.222715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9814843d-9c41-4f9c-b91d-adcd61ed81c2-metrics-certs\") pod \"maas-controller-bbd9d7d98-spcd9\" (UID: \"9814843d-9c41-4f9c-b91d-adcd61ed81c2\") " pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.323301 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.323273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6fs\" (UniqueName: \"kubernetes.io/projected/9814843d-9c41-4f9c-b91d-adcd61ed81c2-kube-api-access-rb6fs\") pod \"maas-controller-bbd9d7d98-spcd9\" (UID: \"9814843d-9c41-4f9c-b91d-adcd61ed81c2\") " pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.323426 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.323306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9814843d-9c41-4f9c-b91d-adcd61ed81c2-metrics-certs\") pod \"maas-controller-bbd9d7d98-spcd9\" (UID: \"9814843d-9c41-4f9c-b91d-adcd61ed81c2\") " pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.323426 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.323383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9814843d-9c41-4f9c-b91d-adcd61ed81c2-webhook-cert\") pod \"maas-controller-bbd9d7d98-spcd9\" (UID: \"9814843d-9c41-4f9c-b91d-adcd61ed81c2\") " pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.325729 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.325706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9814843d-9c41-4f9c-b91d-adcd61ed81c2-webhook-cert\") pod \"maas-controller-bbd9d7d98-spcd9\" (UID: \"9814843d-9c41-4f9c-b91d-adcd61ed81c2\") " pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.325835 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.325741 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9814843d-9c41-4f9c-b91d-adcd61ed81c2-metrics-certs\") pod \"maas-controller-bbd9d7d98-spcd9\" (UID: \"9814843d-9c41-4f9c-b91d-adcd61ed81c2\") " pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.331406 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.331385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6fs\" (UniqueName: \"kubernetes.io/projected/9814843d-9c41-4f9c-b91d-adcd61ed81c2-kube-api-access-rb6fs\") pod \"maas-controller-bbd9d7d98-spcd9\" (UID: \"9814843d-9c41-4f9c-b91d-adcd61ed81c2\") " pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.434149 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.434126 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:14.705025 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:14.705000 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbd9d7d98-spcd9"] Sep 16 08:10:14.706501 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:10:14.706475 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9814843d_9c41_4f9c_b91d_adcd61ed81c2.slice/crio-87cf6d34ec5ca14991c427ba2cd03b82b32d12d41cb97fec67f097a35ffc2318 WatchSource:0}: Error finding container 87cf6d34ec5ca14991c427ba2cd03b82b32d12d41cb97fec67f097a35ffc2318: Status 404 returned error can't find the container with id 87cf6d34ec5ca14991c427ba2cd03b82b32d12d41cb97fec67f097a35ffc2318 Sep 16 08:10:15.258762 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:15.258725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd9d7d98-spcd9" event={"ID":"9814843d-9c41-4f9c-b91d-adcd61ed81c2","Type":"ContainerStarted","Data":"0a3b0d3a53a0bf867f89b869280dfdff1e5b1b5f19ba7c8bec18b67a56046df7"} Sep 16 08:10:15.259188 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:15.258779 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd9d7d98-spcd9" event={"ID":"9814843d-9c41-4f9c-b91d-adcd61ed81c2","Type":"ContainerStarted","Data":"87cf6d34ec5ca14991c427ba2cd03b82b32d12d41cb97fec67f097a35ffc2318"} Sep 16 08:10:15.259188 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:15.258843 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:15.278742 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:15.278682 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bbd9d7d98-spcd9" podStartSLOduration=0.955085921 podStartE2EDuration="1.278666406s" totalImagesPullingTime="323.580485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:10:15.276332835 +0000 UTC m=+2188.195948782" watchObservedRunningTime="2026-09-16 08:10:15.278666406 +0000 UTC m=+2188.198282390" Sep 16 08:10:26.267271 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:26.267233 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bbd9d7d98-spcd9" Sep 16 08:10:38.308964 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:38.308931 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:10:46.205320 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:46.205286 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:10:54.003140 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:10:54.003104 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:11:02.505233 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:11:02.505178 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:11:09.905068 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:11:09.905031 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:11:18.403902 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:11:18.403870 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:11:27.399096 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:11:27.399059 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:11:35.897967 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:11:35.897937 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:12:59.907056 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:12:59.906983 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:13:06.708199 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:13:06.708162 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:13:14.204278 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:13:14.204241 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:13:26.613479 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:13:26.613439 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:13:42.825338 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:13:42.825297 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:13:47.723168 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:13:47.723136 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 08:13:47.731314 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:13:47.731281 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 08:14:23.815651 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:23.815603 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:14:32.901770 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:32.901730 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:14:42.204279 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.204240 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:14:42.445682 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.445643 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs"] Sep 16 08:14:42.449049 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.449026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.460484 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.460457 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs"] Sep 16 08:14:42.610112 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.610079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56215b73-7f52-41e8-96b1-aaaec3bb3996-openshift-service-ca\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.610112 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.610112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56215b73-7f52-41e8-96b1-aaaec3bb3996-metrics-certs\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.610332 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.610143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9vm\" (UniqueName: \"kubernetes.io/projected/56215b73-7f52-41e8-96b1-aaaec3bb3996-kube-api-access-7v9vm\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.610332 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.610265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56215b73-7f52-41e8-96b1-aaaec3bb3996-maas-api-tls\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.711446 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.711377 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56215b73-7f52-41e8-96b1-aaaec3bb3996-maas-api-tls\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.711446 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.711432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56215b73-7f52-41e8-96b1-aaaec3bb3996-openshift-service-ca\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.711609 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.711448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56215b73-7f52-41e8-96b1-aaaec3bb3996-metrics-certs\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.711609 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.711488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9vm\" (UniqueName: \"kubernetes.io/projected/56215b73-7f52-41e8-96b1-aaaec3bb3996-kube-api-access-7v9vm\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.712223 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.712179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56215b73-7f52-41e8-96b1-aaaec3bb3996-openshift-service-ca\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.713993 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.713967 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56215b73-7f52-41e8-96b1-aaaec3bb3996-metrics-certs\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.714076 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.714016 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56215b73-7f52-41e8-96b1-aaaec3bb3996-maas-api-tls\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.719504 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.719484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9vm\" (UniqueName: \"kubernetes.io/projected/56215b73-7f52-41e8-96b1-aaaec3bb3996-kube-api-access-7v9vm\") pod \"maas-api-b85d5f948-nxdhs\" (UID: \"56215b73-7f52-41e8-96b1-aaaec3bb3996\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:42.761150 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:42.761124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:43.038971 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:43.038945 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs"] Sep 16 08:14:43.041670 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:14:43.041643 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56215b73_7f52_41e8_96b1_aaaec3bb3996.slice/crio-001c9eece5f363f1001ccf255a7ba986c283b2cf7824cedc09e7921f07d5941b WatchSource:0}: Error finding container 001c9eece5f363f1001ccf255a7ba986c283b2cf7824cedc09e7921f07d5941b: Status 404 returned error can't find the container with id 001c9eece5f363f1001ccf255a7ba986c283b2cf7824cedc09e7921f07d5941b Sep 16 08:14:43.735655 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:43.735567 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" event={"ID":"56215b73-7f52-41e8-96b1-aaaec3bb3996","Type":"ContainerStarted","Data":"0cd023e9146fd5b7d51f12b94d1fff08ab45479e64347bc743242ffaeff54643"} Sep 16 08:14:43.735655 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:43.735628 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" event={"ID":"56215b73-7f52-41e8-96b1-aaaec3bb3996","Type":"ContainerStarted","Data":"001c9eece5f363f1001ccf255a7ba986c283b2cf7824cedc09e7921f07d5941b"} Sep 16 08:14:43.736056 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:43.735667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:43.753585 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:43.753520 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" podStartSLOduration=1.336237193 podStartE2EDuration="1.75350637s" totalImagesPullingTime="417.269177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:14:43.752265444 +0000 UTC m=+2456.671881391" watchObservedRunningTime="2026-09-16 08:14:43.75350637 +0000 UTC m=+2456.673122316" Sep 16 08:14:49.746104 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:49.746072 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b85d5f948-nxdhs" Sep 16 08:14:50.108418 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:50.108339 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:14:59.716424 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:59.716384 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:14:59.738909 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:59.738878 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9"] Sep 16 08:14:59.739106 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:14:59.739075 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" podUID="649a9983-4492-428b-9e73-bc09c7042fbb" containerName="maas-api" containerID="cri-o://649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155" gracePeriod=30 Sep 16 08:15:00.982224 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:00.982193 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 08:15:01.163657 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.163628 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrk68\" (UniqueName: \"kubernetes.io/projected/649a9983-4492-428b-9e73-bc09c7042fbb-kube-api-access-xrk68\") pod \"649a9983-4492-428b-9e73-bc09c7042fbb\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " Sep 16 08:15:01.163847 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.163832 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/649a9983-4492-428b-9e73-bc09c7042fbb-openshift-service-ca\") pod \"649a9983-4492-428b-9e73-bc09c7042fbb\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " Sep 16 08:15:01.163920 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.163858 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-maas-api-tls\") pod \"649a9983-4492-428b-9e73-bc09c7042fbb\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " Sep 16 08:15:01.163920 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.163880 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-metrics-certs\") pod \"649a9983-4492-428b-9e73-bc09c7042fbb\" (UID: \"649a9983-4492-428b-9e73-bc09c7042fbb\") " Sep 16 08:15:01.164277 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.164254 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649a9983-4492-428b-9e73-bc09c7042fbb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "649a9983-4492-428b-9e73-bc09c7042fbb" (UID: "649a9983-4492-428b-9e73-bc09c7042fbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:15:01.165782 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.165759 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649a9983-4492-428b-9e73-bc09c7042fbb-kube-api-access-xrk68" (OuterVolumeSpecName: "kube-api-access-xrk68") pod "649a9983-4492-428b-9e73-bc09c7042fbb" (UID: "649a9983-4492-428b-9e73-bc09c7042fbb"). InnerVolumeSpecName "kube-api-access-xrk68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:15:01.165916 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.165898 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "649a9983-4492-428b-9e73-bc09c7042fbb" (UID: "649a9983-4492-428b-9e73-bc09c7042fbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:15:01.166016 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.165992 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "649a9983-4492-428b-9e73-bc09c7042fbb" (UID: "649a9983-4492-428b-9e73-bc09c7042fbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:15:01.264663 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.264627 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/649a9983-4492-428b-9e73-bc09c7042fbb-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:15:01.264663 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.264662 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:15:01.264844 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.264675 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/649a9983-4492-428b-9e73-bc09c7042fbb-metrics-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:15:01.264844 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.264686 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrk68\" (UniqueName: \"kubernetes.io/projected/649a9983-4492-428b-9e73-bc09c7042fbb-kube-api-access-xrk68\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Sep 16 08:15:01.837384 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.837300 2580 generic.go:356] "Generic (PLEG): container finished" podID="649a9983-4492-428b-9e73-bc09c7042fbb" containerID="649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155" exitCode=0 Sep 16 08:15:01.837384 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.837348 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" Sep 16 08:15:01.837384 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.837366 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" event={"ID":"649a9983-4492-428b-9e73-bc09c7042fbb","Type":"ContainerDied","Data":"649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155"} Sep 16 08:15:01.837615 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.837424 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9" event={"ID":"649a9983-4492-428b-9e73-bc09c7042fbb","Type":"ContainerDied","Data":"bba778f4c5524d468a2cd06740f32c7244efb00bbe0a2381564f983d3017d626"} Sep 16 08:15:01.837615 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.837455 2580 scope.go:117] "RemoveContainer" containerID="649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155" Sep 16 08:15:01.846573 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.846549 2580 scope.go:117] "RemoveContainer" containerID="649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155" Sep 16 08:15:01.846815 ip-10-0-131-134 kubenswrapper[2580]: E0916 08:15:01.846797 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155\": container with ID starting with 649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155 not found: ID does not exist" containerID="649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155" Sep 16 08:15:01.846886 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.846823 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155"} err="failed to get container status \"649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155\": rpc error: code = NotFound desc = could not find container \"649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155\": container with ID starting with 649b33d6f9304fc2f4809a184367d52d628f293ea854b18e17636425409b3155 not found: ID does not exist" Sep 16 08:15:01.854600 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.854575 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9"] Sep 16 08:15:01.858177 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:01.858152 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-55gz9"] Sep 16 08:15:03.560171 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:03.560140 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="649a9983-4492-428b-9e73-bc09c7042fbb" path="/var/lib/kubelet/pods/649a9983-4492-428b-9e73-bc09c7042fbb/volumes" Sep 16 08:15:05.709448 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:05.709414 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:15:13.303499 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:13.303463 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6tgn9"] Sep 16 08:15:15.392493 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:15.392456 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58458bb776-79rkb_df2f5e81-9bba-4cd7-afdf-956188daa9ed/authorino/0.log" Sep 16 08:15:20.878270 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:20.878234 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bbd9d7d98-spcd9_9814843d-9c41-4f9c-b91d-adcd61ed81c2/manager/0.log" Sep 16 08:15:21.199701 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:21.199676 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-gcqhh_71007db5-1d66-4e57-af11-cfc63b4467b7/copy-manifests/0.log" Sep 16 08:15:21.205600 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:21.205577 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-gcqhh_71007db5-1d66-4e57-af11-cfc63b4467b7/manager/0.log" Sep 16 08:15:21.802596 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:21.802567 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-b85d5f948-nxdhs_56215b73-7f52-41e8-96b1-aaaec3bb3996/maas-api/0.log" Sep 16 08:15:22.335625 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:22.335597 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-682z2_75c1db6e-0e1a-4e55-9636-e1c826ffc392/postgres/0.log" Sep 16 08:15:23.089411 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:23.089378 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58458bb776-79rkb_df2f5e81-9bba-4cd7-afdf-956188daa9ed/authorino/0.log" Sep 16 08:15:23.311023 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:23.310990 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-4vfll_27748d8c-eb94-4e23-977d-62c8314595f0/manager/0.log" Sep 16 08:15:23.649297 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:23.649266 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6tgn9_61ca0c67-45eb-4fe9-965c-eee043f7c7fe/limitador/0.log" Sep 16 08:15:23.763169 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:23.763138 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-8q5lx_cb7a369c-acbf-4bbd-96b8-3a5e3df15c58/manager/0.log" Sep 16 08:15:24.777281 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:24.777254 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fh9w6_945b9e69-a7b4-458a-a638-5a9ffcc8cd8e/payload-processing/0.log" Sep 16 08:15:24.879515 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:24.879492 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58f8796bd8-45bdh_60e1aabf-d4f3-4f85-83dc-72c8c5c61987/router/0.log" Sep 16 08:15:30.973568 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:30.973536 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxrzq/must-gather-6zngr"] Sep 16 08:15:30.974019 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:30.973917 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="649a9983-4492-428b-9e73-bc09c7042fbb" containerName="maas-api" Sep 16 08:15:30.974019 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:30.973927 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="649a9983-4492-428b-9e73-bc09c7042fbb" containerName="maas-api" Sep 16 08:15:30.974019 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:30.973995 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="649a9983-4492-428b-9e73-bc09c7042fbb" containerName="maas-api" Sep 16 08:15:30.977001 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:30.976984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxrzq/must-gather-6zngr" Sep 16 08:15:30.979869 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:30.979841 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zxrzq\"/\"default-dockercfg-nr2xl\"" Sep 16 08:15:30.980021 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:30.979951 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxrzq\"/\"kube-root-ca.crt\"" Sep 16 08:15:30.980253 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:30.980234 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxrzq\"/\"openshift-service-ca.crt\"" Sep 16 08:15:30.986794 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:30.986776 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxrzq/must-gather-6zngr"] Sep 16 08:15:31.015797 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:31.015764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfc7j\" (UniqueName: \"kubernetes.io/projected/e616258a-7010-49d8-9d46-dcca21af0f0d-kube-api-access-pfc7j\") pod \"must-gather-6zngr\" (UID: \"e616258a-7010-49d8-9d46-dcca21af0f0d\") " pod="openshift-must-gather-zxrzq/must-gather-6zngr" Sep 16 08:15:31.015934 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:31.015828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e616258a-7010-49d8-9d46-dcca21af0f0d-must-gather-output\") pod \"must-gather-6zngr\" (UID: \"e616258a-7010-49d8-9d46-dcca21af0f0d\") " pod="openshift-must-gather-zxrzq/must-gather-6zngr" Sep 16 08:15:31.117018 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:31.116989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e616258a-7010-49d8-9d46-dcca21af0f0d-must-gather-output\") pod \"must-gather-6zngr\" (UID: \"e616258a-7010-49d8-9d46-dcca21af0f0d\") " pod="openshift-must-gather-zxrzq/must-gather-6zngr" Sep 16 08:15:31.117128 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:31.117067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfc7j\" (UniqueName: \"kubernetes.io/projected/e616258a-7010-49d8-9d46-dcca21af0f0d-kube-api-access-pfc7j\") pod \"must-gather-6zngr\" (UID: \"e616258a-7010-49d8-9d46-dcca21af0f0d\") " pod="openshift-must-gather-zxrzq/must-gather-6zngr" Sep 16 08:15:31.117367 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:31.117349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e616258a-7010-49d8-9d46-dcca21af0f0d-must-gather-output\") pod \"must-gather-6zngr\" (UID: \"e616258a-7010-49d8-9d46-dcca21af0f0d\") " pod="openshift-must-gather-zxrzq/must-gather-6zngr" Sep 16 08:15:31.126875 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:31.126846 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfc7j\" (UniqueName: \"kubernetes.io/projected/e616258a-7010-49d8-9d46-dcca21af0f0d-kube-api-access-pfc7j\") pod \"must-gather-6zngr\" (UID: \"e616258a-7010-49d8-9d46-dcca21af0f0d\") " pod="openshift-must-gather-zxrzq/must-gather-6zngr" Sep 16 08:15:31.288066 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:31.287997 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxrzq/must-gather-6zngr" Sep 16 08:15:31.561173 ip-10-0-131-134 kubenswrapper[2580]: W0916 08:15:31.561143 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode616258a_7010_49d8_9d46_dcca21af0f0d.slice/crio-44ffa5b9692a8a4f48da082316d4bc287eb23f0ee9ca7cbbd112f94a3911e504 WatchSource:0}: Error finding container 44ffa5b9692a8a4f48da082316d4bc287eb23f0ee9ca7cbbd112f94a3911e504: Status 404 returned error can't find the container with id 44ffa5b9692a8a4f48da082316d4bc287eb23f0ee9ca7cbbd112f94a3911e504 Sep 16 08:15:31.562715 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:31.562690 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxrzq/must-gather-6zngr"] Sep 16 08:15:31.562874 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:31.562847 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 08:15:32.009288 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:32.009252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxrzq/must-gather-6zngr" event={"ID":"e616258a-7010-49d8-9d46-dcca21af0f0d","Type":"ContainerStarted","Data":"44ffa5b9692a8a4f48da082316d4bc287eb23f0ee9ca7cbbd112f94a3911e504"} Sep 16 08:15:33.020306 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:33.020265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxrzq/must-gather-6zngr" event={"ID":"e616258a-7010-49d8-9d46-dcca21af0f0d","Type":"ContainerStarted","Data":"de7cf981edba5aba5824f9015b4cf2309f4122279f4c7c2d7a0cb9d3ed9e6b62"} Sep 16 08:15:33.020852 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:33.020828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxrzq/must-gather-6zngr" event={"ID":"e616258a-7010-49d8-9d46-dcca21af0f0d","Type":"ContainerStarted","Data":"b2c8d3acaac051f71ecffa74181e14d06149f5401ff3326d2e49456373005879"} Sep 16 08:15:33.921522 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:33.921494 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l2vjc_b40f3f8d-04c4-4581-b929-fabed41b7079/global-pull-secret-syncer/0.log" Sep 16 08:15:34.103232 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:34.103188 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wscpg_2270b6d5-4d64-43d1-8130-c30279ab526b/konnectivity-agent/0.log" Sep 16 08:15:34.125362 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:34.125328 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-134.ec2.internal_f3db43c1838ed8ed4417671dad4e9e6c/haproxy/0.log" Sep 16 08:15:38.427796 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:38.427761 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58458bb776-79rkb_df2f5e81-9bba-4cd7-afdf-956188daa9ed/authorino/0.log" Sep 16 08:15:38.548335 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:38.548309 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-4vfll_27748d8c-eb94-4e23-977d-62c8314595f0/manager/0.log" Sep 16 08:15:38.711322 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:38.711237 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6tgn9_61ca0c67-45eb-4fe9-965c-eee043f7c7fe/limitador/0.log" Sep 16 08:15:38.787535 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:38.787499 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-8q5lx_cb7a369c-acbf-4bbd-96b8-3a5e3df15c58/manager/0.log" Sep 16 08:15:40.467449 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:40.467355 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-sppbq_5cfd20db-3fe7-47a1-bb11-8a8421cd83fb/cluster-monitoring-operator/0.log" Sep 16 08:15:40.493078 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:40.493043 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fhw4m_d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5/kube-state-metrics/0.log" Sep 16 08:15:40.513121 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:40.513073 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fhw4m_d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5/kube-rbac-proxy-main/0.log" Sep 16 08:15:40.540619 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:40.540573 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fhw4m_d0351e2b-2cff-4db7-9e0f-b5cbae9f4bb5/kube-rbac-proxy-self/0.log" Sep 16 08:15:40.569078 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:40.569051 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-584ff4576-m57c9_40595f6f-bce1-42a4-ad70-d5591b98bd9e/metrics-server/0.log" Sep 16 08:15:40.632614 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:40.632579 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8prhl_3a4c6bf0-be34-4bbc-8473-246793553bda/node-exporter/0.log" Sep 16 08:15:40.652639 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:40.652600 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8prhl_3a4c6bf0-be34-4bbc-8473-246793553bda/kube-rbac-proxy/0.log" Sep 16 08:15:40.679381 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:40.679339 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8prhl_3a4c6bf0-be34-4bbc-8473-246793553bda/init-textfile/0.log" Sep 16 08:15:41.242033 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:41.242006 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54889c48b5-pbxts_9fc426e4-6f37-4064-884c-0b6e92572f18/thanos-query/0.log" Sep 16 08:15:41.263125 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:41.263098 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54889c48b5-pbxts_9fc426e4-6f37-4064-884c-0b6e92572f18/kube-rbac-proxy-web/0.log" Sep 16 08:15:41.284067 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:41.284002 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54889c48b5-pbxts_9fc426e4-6f37-4064-884c-0b6e92572f18/kube-rbac-proxy/0.log" Sep 16 08:15:41.304751 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:41.304723 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54889c48b5-pbxts_9fc426e4-6f37-4064-884c-0b6e92572f18/prom-label-proxy/0.log" Sep 16 08:15:41.326383 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:41.326334 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54889c48b5-pbxts_9fc426e4-6f37-4064-884c-0b6e92572f18/kube-rbac-proxy-rules/0.log" Sep 16 08:15:41.355398 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:41.355376 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54889c48b5-pbxts_9fc426e4-6f37-4064-884c-0b6e92572f18/kube-rbac-proxy-metrics/0.log" Sep 16 08:15:42.920144 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:42.920110 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-cvm5x_6c06db71-a2bf-4b00-82f2-3c1bc21555a0/networking-console-plugin/0.log" Sep 16 08:15:43.516359 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:43.516326 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/2.log" Sep 16 08:15:43.516934 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:43.516908 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-d5hqp_0a3ad689-9905-4ae0-8ad1-0364dea219fe/console-operator/1.log" Sep 16 08:15:44.044381 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.044342 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78b985f7-9kmpr_b6d55c77-37f6-4853-9190-a185eb956de3/console/0.log" Sep 16 08:15:44.549503 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.549423 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zxrzq/must-gather-6zngr" podStartSLOduration=13.704756987 podStartE2EDuration="14.549400299s" totalImagesPullingTime="844.643312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:15:33.039730624 +0000 UTC m=+2505.959346572" watchObservedRunningTime="2026-09-16 08:15:44.549400299 +0000 UTC m=+2517.469016245" Sep 16 08:15:44.552302 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.552270 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt"] Sep 16 08:15:44.560875 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.560846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.565440 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.565415 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt"] Sep 16 08:15:44.666969 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.666917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-lib-modules\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.667155 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.667006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-sys\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.667155 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.667074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-podres\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.667155 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.667100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4z6z\" (UniqueName: \"kubernetes.io/projected/537b3330-5150-4bb5-aa7c-b882bccd5289-kube-api-access-h4z6z\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.667362 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.667266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-proc\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.768237 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.768165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-lib-modules\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.768422 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.768259 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-sys\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.768422 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.768305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-podres\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.768422 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.768333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4z6z\" (UniqueName: \"kubernetes.io/projected/537b3330-5150-4bb5-aa7c-b882bccd5289-kube-api-access-h4z6z\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.768422 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.768407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-proc\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.768732 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.768534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-proc\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.768732 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.768666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-lib-modules\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.768732 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.768721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-sys\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.768900 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.768799 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/537b3330-5150-4bb5-aa7c-b882bccd5289-podres\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.770988 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.770958 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-p8wns_7b73b576-b7d6-46ec-a73c-d6cdc1ec0fc7/volume-data-source-validator/0.log" Sep 16 08:15:44.780181 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.780150 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4z6z\" (UniqueName: \"kubernetes.io/projected/537b3330-5150-4bb5-aa7c-b882bccd5289-kube-api-access-h4z6z\") pod \"perf-node-gather-daemonset-slfvt\" (UID: \"537b3330-5150-4bb5-aa7c-b882bccd5289\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:44.875850 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:44.875530 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:45.173936 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:45.173903 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt"] Sep 16 08:15:45.950897 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:45.950863 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2pxc6_3d785aaa-20ec-484a-a81e-48247bc83110/dns/0.log" Sep 16 08:15:45.971665 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:45.971636 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2pxc6_3d785aaa-20ec-484a-a81e-48247bc83110/kube-rbac-proxy/0.log" Sep 16 08:15:46.077437 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:46.077410 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h69ht_5b2f9240-4d84-45e6-bfc3-fa31321bda0c/dns-node-resolver/0.log" Sep 16 08:15:46.133618 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:46.133569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" event={"ID":"537b3330-5150-4bb5-aa7c-b882bccd5289","Type":"ContainerStarted","Data":"845a23c90a6fe9445b1e7316917f5e697a9165eaa7cb77694f9f22e63b4f9665"} Sep 16 08:15:46.133921 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:46.133643 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:46.133921 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:46.133669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" event={"ID":"537b3330-5150-4bb5-aa7c-b882bccd5289","Type":"ContainerStarted","Data":"eda9b7a422d924b66a3723e33b65b2d540e8452996304d6d1dbcd4f042dee863"} Sep 16 08:15:46.153812 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:46.153751 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" podStartSLOduration=2.153729395 podStartE2EDuration="2.153729395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 08:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:15:46.150826919 +0000 UTC m=+2519.070442866" watchObservedRunningTime="2026-09-16 08:15:46.153729395 +0000 UTC m=+2519.073345342" Sep 16 08:15:46.636813 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:46.636781 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9w78q_36dbd8bf-bd64-4f3c-9d7f-ab618f797a96/node-ca/0.log" Sep 16 08:15:48.100163 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:48.099831 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fh9w6_945b9e69-a7b4-458a-a638-5a9ffcc8cd8e/payload-processing/0.log" Sep 16 08:15:48.123632 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:48.123605 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58f8796bd8-45bdh_60e1aabf-d4f3-4f85-83dc-72c8c5c61987/router/0.log" Sep 16 08:15:49.126926 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:49.126890 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-85jjc_0b6f5a2d-a0c2-47bd-886c-7a68b8c9bd9b/serve-healthcheck-canary/0.log" Sep 16 08:15:49.725132 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:49.725096 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-vfmqh_3a8af6d2-1836-4dc6-81c3-85fc2466ba2d/insights-operator/0.log" Sep 16 08:15:49.725583 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:49.725561 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-vfmqh_3a8af6d2-1836-4dc6-81c3-85fc2466ba2d/insights-operator/1.log" Sep 16 08:15:49.817585 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:49.817556 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j28rh_73507fa6-1ed8-49e8-ab6c-6f635b78a0a0/kube-rbac-proxy/0.log" Sep 16 08:15:49.838136 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:49.838094 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j28rh_73507fa6-1ed8-49e8-ab6c-6f635b78a0a0/exporter/0.log" Sep 16 08:15:49.860319 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:49.860284 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j28rh_73507fa6-1ed8-49e8-ab6c-6f635b78a0a0/extractor/0.log" Sep 16 08:15:52.153236 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:52.153177 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-slfvt" Sep 16 08:15:52.360083 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:52.360052 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bbd9d7d98-spcd9_9814843d-9c41-4f9c-b91d-adcd61ed81c2/manager/0.log" Sep 16 08:15:52.561689 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:52.561613 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-gcqhh_71007db5-1d66-4e57-af11-cfc63b4467b7/manager/0.log" Sep 16 08:15:52.581749 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:52.581727 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-gcqhh_71007db5-1d66-4e57-af11-cfc63b4467b7/copy-manifests/0.log" Sep 16 08:15:54.114002 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:54.113969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f78fc697-jw98p_e58dec74-9f8d-409a-9c83-69b2150eb575/manager/0.log" Sep 16 08:15:54.139850 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:54.139806 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-xsxqg_00c65999-a0af-4804-8c78-99171cd155f8/openshift-lws-operator/0.log" Sep 16 08:15:55.296677 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:55.296639 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fh9w6_945b9e69-a7b4-458a-a638-5a9ffcc8cd8e/payload-processing/0.log" Sep 16 08:15:55.309037 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:55.309008 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58f8796bd8-45bdh_60e1aabf-d4f3-4f85-83dc-72c8c5c61987/router/0.log" Sep 16 08:15:57.368850 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:57.368819 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-fh9w6_945b9e69-a7b4-458a-a638-5a9ffcc8cd8e/payload-processing/0.log" Sep 16 08:15:57.382453 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:57.382428 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58f8796bd8-45bdh_60e1aabf-d4f3-4f85-83dc-72c8c5c61987/router/0.log" Sep 16 08:15:59.842274 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:59.842204 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9rgl4_6f8ff03a-7366-4924-a639-ec3d9e63c398/kube-storage-version-migrator-operator/1.log" Sep 16 08:15:59.843187 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:15:59.843159 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9rgl4_6f8ff03a-7366-4924-a639-ec3d9e63c398/kube-storage-version-migrator-operator/0.log" Sep 16 08:16:00.760660 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:16:00.760632 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pl84_05675886-b0f7-47ae-b1ee-355badf36fff/kube-multus-additional-cni-plugins/0.log" Sep 16 08:16:00.780287 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:16:00.780257 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pl84_05675886-b0f7-47ae-b1ee-355badf36fff/egress-router-binary-copy/0.log" Sep 16 08:16:00.799428 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:16:00.799406 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pl84_05675886-b0f7-47ae-b1ee-355badf36fff/cni-plugins/0.log" Sep 16 08:16:00.817133 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:16:00.817111 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pl84_05675886-b0f7-47ae-b1ee-355badf36fff/bond-cni-plugin/0.log" Sep 16 08:16:00.834891 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:16:00.834869 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pl84_05675886-b0f7-47ae-b1ee-355badf36fff/routeoverride-cni/0.log" Sep 16 08:16:00.852138 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:16:00.852114 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pl84_05675886-b0f7-47ae-b1ee-355badf36fff/whereabouts-cni-bincopy/0.log" Sep 16 08:16:00.871551 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:16:00.871530 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pl84_05675886-b0f7-47ae-b1ee-355badf36fff/whereabouts-cni/0.log" Sep 16 08:16:01.271413 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:16:01.271384 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zz4tf_8065977b-91e5-4fed-8250-5d4f379c05e3/kube-multus/0.log" Sep 16 08:16:01.291486 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:16:01.291453 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8qm8r_533591c0-bcd3-4f1f-9ba5-3eb2effa48a1/network-metrics-daemon/0.log" Sep 16 08:16:01.304991 ip-10-0-131-134 kubenswrapper[2580]: I0916 08:16:01.304973 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8qm8r_533591c0-bcd3-4f1f-9ba5-3eb2effa48a1/kube-rbac-proxy/0.log"